2016-05-18 00:57:18,767 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(197)) - found 1 test platforms from classpath
2016-05-18 00:57:18,801 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(246)) - adding test: cascading.detail.EveryPipeAssemblyPlatformTest, with platform: hadoop2-tez
2016-05-18 00:57:19,177 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-05-18 00:57:19,660 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-18 00:57:19,972 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(75)) - platform property overrides:
2016-05-18 00:57:20,202 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:20,204 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:20,336 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: C3AC9435A9764C66BCA87D0F68AD97F8
2016-05-18 00:57:20,525 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.080
2016-05-18 00:57:20,542 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.098
2016-05-18 00:57:20,543 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:20,544 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:20,545 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:20,706 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:20,839 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_-1
2016-05-18 00:57:21,066 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.1.0
2016-05-18 00:57:21,069 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:21,070 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:21,071 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_-1"]
2016-05-18 00:57:21,071 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:21,072 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:21,072 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:21,073 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...blyplatform/-1_'field'_-1
2016-05-18 00:57:21,074 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:21,104 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:21,275 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:21,286 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:21,286 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:21,363 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001 doesn't exist and is created
2016-05-18 00:57:21,455 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:21,455 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:21,457 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001_wd
2016-05-18 00:57:21,526 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533041285_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:21,541 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:21,542 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:21,554 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:21,554 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:21,554 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:21,555 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:21,556 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-18 00:57:21,556 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-18 00:57:21,556 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:21,584 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:21,614 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:21,615 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:21,617 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:21,618 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:21,619 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:21,619 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:21,620 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:21,621 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:21,622 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:21,623 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:21,635 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:21,641 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:21,657 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-18 00:57:21,657 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-18 00:57:21,657 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:21,671 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:21,673 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:21,674 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:21,678 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001_wd/localmode-log-dir
2016-05-18 00:57:21,679 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001_wd/localmode-log-dir/history.txt.appattempt_1463533041285_0001_000000, maxErrors=10
2016-05-18 00:57:21,679 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:21,682 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533041285_0001_000000, appSubmitTime=1463533041504, launchTime=1463533041524
2016-05-18 00:57:21,690 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:21,749 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:56375
2016-05-18 00:57:21,751 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533041285
2016-05-18 00:57:21,757 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-05-18 00:57:21,758 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-05-18 00:57:21,758 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:21,771 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533041285_0001_000000, startTime=1463533041770
2016-05-18 00:57:21,771 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:21,858 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:21,877 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:21,877 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...blyplatform/-1_'field'_-1, applicationId=application_1463533041285_0001, dagName=[5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443] every/(1/1) ...blyplatform/-1_'field'_-1
2016-05-18 00:57:22,125 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443] every/(1/1) ...blyplatform/-1_'field'_-1
2016-05-18 00:57:22,133 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:22,203 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533041285_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001_wd/localmode-log-dir/dag_1463533041285_0001_1.dot
2016-05-18 00:57:22,209 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001_wd/localmode-log-dir/dag_1463533041285_0001_1-tez-dag.pb.txt
2016-05-18 00:57:22,266 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533041285_0001_1
2016-05-18 00:57:22,266 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:22,267 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443] every/(1/1) ...blyplatform/-1_'field'_-1
2016-05-18 00:57:22 Running Dag: dag_1463533041285_0001_1
2016-05-18 00:57:22,291 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533041285_0001_1, submitTime=1463533042125
2016-05-18 00:57:22,356 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001/tez-conf.pb" } size: 36913 timestamp: 1463533041000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:22,357 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,359 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:22,361 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001/tez-conf.pb" } size: 36913 timestamp: 1463533041000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:22,361 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]
2016-05-18 00:57:22,363 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:22,374 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:22,379 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533041285_0001_1, initTime=1463533042378
2016-05-18 00:57:22,379 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533041285_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:22,381 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:22,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:DAG_STARTED]: dagID=dag_1463533041285_0001_1, startTime=1463533042384
2016-05-18 00:57:22,384 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533041285_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:22,385 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E79C51E410E44BC18BA64F45AEAB2D8B : {1FBD4AEB772D4D5897CDA06474D520E3={InputName=1FBD4AEB772D4D5897CDA06474D520E3}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:22,386 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 1FBD4AEB772D4D5897CDA06474D520E3, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:22,386 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,389 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...blyplatform/-1_'field'_-1, applicationId=application_1463533041285_0001, dagName=[5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443] every/(1/1) ...blyplatform/-1_'field'_-1
2016-05-18 00:57:22,394 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,394 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,396 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,402 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 1FBD4AEB772D4D5897CDA06474D520E3 on vertex vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,415 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:22,415 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]
2016-05-18 00:57:22,428 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 1FBD4AEB772D4D5897CDA06474D520E3 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:22,459 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533041285_0001, with dag id: dag_1463533041285_0001_1
2016-05-18 00:57:22,465 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:22,466 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:22,468 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:22,469 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:22,472 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:22,473 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]
2016-05-18 00:57:22,484 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 1FBD4AEB772D4D5897CDA06474D520E3 on vertex vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,491 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]
2016-05-18 00:57:22,492 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533041285_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:22,493 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD], numAdditionalOutputs=1
2016-05-18 00:57:22,494 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=798B4E882BBE4B859073028CC8B85C38, vertex=vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:22,511 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D5AB476D53F14ED293048EBC83CAF3FD:798B4E882BBE4B859073028CC8B85C38 using oldmapred API
2016-05-18 00:57:22,513 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=798B4E882BBE4B859073028CC8B85C38, vertexName=D5AB476D53F14ED293048EBC83CAF3FD, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:22,528 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D5AB476D53F14ED293048EBC83CAF3FD, vertexId=vertex_1463533041285_0001_1_01, initRequestedTime=1463533042415, initedTime=1463533042493, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:22,528 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:22,544 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:22,545 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {1FBD4AEB772D4D5897CDA06474D520E3=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:22,547 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] parallelism set to 1
2016-05-18 00:57:22,547 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:22,558 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E79C51E410E44BC18BA64F45AEAB2D8B, vertexId=vertex_1463533041285_0001_1_00, initRequestedTime=1463533042385, initedTime=1463533042558, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:22,558 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] with distanceFromRoot: 0
2016-05-18 00:57:22,558 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:22,560 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533041285_0001_1_00, startRequestedTime=1463533042415, startedTime=1463533042559
2016-05-18 00:57:22,560 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533041285_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:22,561 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:22,561 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533041285_0001_1_00 for vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD] numStartedSources: 1 numSources: 1
2016-05-18 00:57:22,562 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD] with distanceFromRoot: 1
2016-05-18 00:57:22,565 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533041285_0001_1_01, startRequestedTime=1463533042561, startedTime=1463533042562
2016-05-18 00:57:22,565 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:22,571 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E79C51E410E44BC18BA64F45AEAB2D8B
2016-05-18 00:57:22,577 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D5AB476D53F14ED293048EBC83CAF3FD with 0 source tasks and 1 pending tasks
2016-05-18 00:57:22,579 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,581 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E79C51E410E44BC18BA64F45AEAB2D8B in vertex: D5AB476D53F14ED293048EBC83CAF3FD numBipartiteSourceTasks: 1
2016-05-18 00:57:22,589 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:TASK_STARTED]: vertexName=E79C51E410E44BC18BA64F45AEAB2D8B, taskId=task_1463533041285_0001_1_00_000000, scheduledTime=1463533042588, launchTime=1463533042588
2016-05-18 00:57:22,610 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:22,651 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:22,667 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533041285_0001_00_000001, launchTime=1463533042666
2016-05-18 00:57:22,671 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533041285_0001_00_000001
2016-05-18 00:57:22,676 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533041285_0001_1_00_000000_0] started. Is using containerId: [container_1463533041285_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:22,677 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533041285_0001_00_000001: 1 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533041285_0001_1_00_000000_0
2016-05-18 00:57:22 Starting to run new task attempt: attempt_1463533041285_0001_1_00_000000_0
2016-05-18 00:57:22,679 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533041285_0001_1_00_000000_0
2016-05-18 00:57:22,683 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:22,679 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E79C51E410E44BC18BA64F45AEAB2D8B, taskAttemptId=attempt_1463533041285_0001_1_00_000000_0, startTime=1463533042675, containerId=container_1463533041285_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:22,684 INFO counters.Limits (Limits.java:ensureInitialized(60)) - Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-05-18 00:57:22,684 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:22,696 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443] every/(1/1) ...blyplatform/-1_'field'_-1, VertexName: E79C51E410E44BC18BA64F45AEAB2D8B, VertexParallelism: 1, TaskAttemptID:attempt_1463533041285_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=1FBD4AEB772D4D5897CDA06474D520E3, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D5AB476D53F14ED293048EBC83CAF3FD, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:22,697 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:22,701 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533041285_0001_1_00_000000_0
2016-05-18 00:57:22,706 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533041285_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:22,708 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533041285_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:22,724 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:22,730 INFO input.MRInput (MRInput.java:initialize(440)) - 1FBD4AEB772D4D5897CDA06474D520E3 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:22,731 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 1FBD4AEB772D4D5897CDA06474D520E3
2016-05-18 00:57:22,741 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E79C51E410E44BC18BA64F45AEAB2D8B, ordinal: 0
2016-05-18 00:57:22,742 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E79C51E410E44BC18BA64F45AEAB2D8B, mem on start (mb), free: 201, total: 281, max: 672
2016-05-18 00:57:22,742 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:22,742 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:22,743 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:22,746 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:22,746 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[1FBD4AEB772D4D5897CDA06474D520E3:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D5AB476D53F14ED293048EBC83CAF3FD:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:22,751 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:22,754 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:22,755 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:22,755 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[1FBD4AEB772D4D5897CDA06474D520E3:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D5AB476D53F14ED293048EBC83CAF3FD:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:22,755 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:22,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 1FBD4AEB772D4D5897CDA06474D520E3 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:22,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 1FBD4AEB772D4D5897CDA06474D520E3
2016-05-18 00:57:22,757 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:22,757 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:22,757 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:22,759 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533041285_0001_1_00_000000_0
2016-05-18 00:57:22,780 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:22,781 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 1FBD4AEB772D4D5897CDA06474D520E3 initialized RecordReader from event
2016-05-18 00:57:22,805 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:22,821 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 1FBD4AEB772D4D5897CDA06474D520E3
2016-05-18 00:57:22,822 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 106531C925854ECBAE53C379BAEDF788
2016-05-18 00:57:22,822 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 106531C925854ECBAE53C379BAEDF788
2016-05-18 00:57:22,862 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:22,888 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:22,888 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D5AB476D53F14ED293048EBC83CAF3FD using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@db3cbc9, comparator=cascading.tuple.hadoop.util.TupleComparator@17080026, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:22,894 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D5AB476D53F14ED293048EBC83CAF3FD: , UsingHashComparator=false
2016-05-18 00:57:23,018 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:23,019 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:23,019 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D5AB476D53F14ED293048EBC83CAF3FD: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:23,021 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D5AB476D53F14ED293048EBC83CAF3FD: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:23,027 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:23,029 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E79C51E410E44BC18BA64F45AEAB2D8B, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:57:23,040 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533041285_0001_1_00_000000_0
2016-05-18 00:57:23,040 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E79C51E410E44BC18BA64F45AEAB2D8B, mem on close (mb), free: 89, total: 281, max: 672
2016-05-18 00:57:23,041 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D5AB476D53F14ED293048EBC83CAF3FD: Starting flush of map output
2016-05-18 00:57:23,041 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D5AB476D53F14ED293048EBC83CAF3FD: Span0.length = 20, perItem = 6
2016-05-18 00:57:23,041 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D5AB476D53F14ED293048EBC83CAF3FD: done sorting span=0, length=20, time=0
2016-05-18 00:57:23,043 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D5AB476D53F14ED293048EBC83CAF3FD: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:23,048 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D5AB476D53F14ED293048EBC83CAF3FD: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001_wd/localmode-local-dir/output/attempt_1463533041285_0001_1_00_000000_0_10003_0/file.out
2016-05-18 00:57:23,077 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:23 Completed running task attempt: attempt_1463533041285_0001_1_00_000000_0
2016-05-18 00:57:23,080 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533041285_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:23,081 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533041285_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:23,083 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533041285_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533043029, Process_Duration=11, Process_End_Time=1463533043040, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:23,088 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533041285_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:23,088 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:23,089 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:23,089 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:23,089 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533041285_0001_00_000001
2016-05-18 00:57:23,091 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E79C51E410E44BC18BA64F45AEAB2D8B, taskAttemptId=attempt_1463533041285_0001_1_00_000000_0, creationTime=1463533042605, allocationTime=1463533042659, startTime=1463533042675, finishTime=1463533043087, timeTaken=412, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:23,098 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533041285_0001_1_00_000000_0
2016-05-18 00:57:23,103 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:TASK_FINISHED]: vertexName=E79C51E410E44BC18BA64F45AEAB2D8B, taskId=task_1463533041285_0001_1_00_000000, startTime=1463533042675, finishTime=1463533043101, timeTaken=426, status=SUCCEEDED, successfulAttemptID=attempt_1463533041285_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533043029, Process_Duration=11, Process_End_Time=1463533043040, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:23,117 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:23,119 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:23,136 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:VERTEX_FINISHED]: vertexName=E79C51E410E44BC18BA64F45AEAB2D8B, vertexId=vertex_1463533041285_0001_1_00, initRequestedTime=1463533042385, initedTime=1463533042558, startRequestedTime=1463533042415, startedTime=1463533042559, finishTime=1463533043119, timeTaken=560, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533043029, Process_Duration=11, Process_End_Time=1463533043040, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533042675, firstTasksToStart=[ task_1463533041285_0001_1_00_000000 ], lastTaskFinishTime=1463533043087, lastTasksToFinish=[ task_1463533041285_0001_1_00_000000 ], minTaskDuration=412, maxTaskDuration=412, avgTaskDuration=412.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533041285_0001_1_00_000000 ], longestDurationTasks=[ task_1463533041285_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:23,136 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:23,138 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533041285_0001_00_000001, stoppedTime=1463533043138, exitStatus=0
2016-05-18 00:57:23,139 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD] attempt: attempt_1463533041285_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:23,139 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D5AB476D53F14ED293048EBC83CAF3FD
2016-05-18 00:57:23,141 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:23,142 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:23,142 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533041285_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:23,142 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E79C51E410E44BC18BA64F45AEAB2D8B, destinationVertexName=D5AB476D53F14ED293048EBC83CAF3FD. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:23,143 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]
2016-05-18 00:57:23,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:TASK_STARTED]: vertexName=D5AB476D53F14ED293048EBC83CAF3FD, taskId=task_1463533041285_0001_1_01_000000, scheduledTime=1463533043144, launchTime=1463533043144
2016-05-18 00:57:23,144 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]
2016-05-18 00:57:23,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533041285_0001_00_000002, launchTime=1463533043146
2016-05-18 00:57:23,147 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533041285_0001_00_000002
2016-05-18 00:57:23,148 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533041285_0001_1_01_000000_0] started. Is using containerId: [container_1463533041285_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:23,148 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D5AB476D53F14ED293048EBC83CAF3FD, taskAttemptId=attempt_1463533041285_0001_1_01_000000_0, startTime=1463533043148, containerId=container_1463533041285_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:23,149 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533041285_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533041285_0001_1_01_000000_0
2016-05-18 00:57:23 Starting to run new task attempt: attempt_1463533041285_0001_1_01_000000_0
2016-05-18 00:57:23,150 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533041285_0001_1_01_000000_0
2016-05-18 00:57:23,151 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:23,151 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:23,151 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443] every/(1/1) ...blyplatform/-1_'field'_-1, VertexName: D5AB476D53F14ED293048EBC83CAF3FD, VertexParallelism: 1, TaskAttemptID:attempt_1463533041285_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E79C51E410E44BC18BA64F45AEAB2D8B, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=798B4E882BBE4B859073028CC8B85C38, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:23,152 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:23,153 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533041285_0001_1_01_000000_0
2016-05-18 00:57:23,155 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533041285_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]
2016-05-18 00:57:23,154 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533041285_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:23,156 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533041285_0001_00_000001: 64 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:23,156 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533041285_0001_00_000001, Exiting
2016-05-18 00:57:23,156 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533041285_0001_00_000001
2016-05-18 00:57:23,156 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533041285_0001_00_000001
2016-05-18 00:57:23,156 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533041285_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:23,165 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:23,173 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533041285_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:23,181 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533041285_0001_00_000001 completed successfully
2016-05-18 00:57:23,187 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D5AB476D53F14ED293048EBC83CAF3FD, ordinal: 1
2016-05-18 00:57:23,187 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D5AB476D53F14ED293048EBC83CAF3FD, mem on start (mb), free: 77, total: 281, max: 672
2016-05-18 00:57:23,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:23,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:23,195 INFO output.MROutput (MROutput.java:initialize(421)) - 798B4E882BBE4B859073028CC8B85C38: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:23,196 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:23,196 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:23,198 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:23,199 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[798B4E882BBE4B859073028CC8B85C38:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [E79C51E410E44BC18BA64F45AEAB2D8B:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:23,199 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:23,199 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:23,200 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:23,200 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[798B4E882BBE4B859073028CC8B85C38:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [E79C51E410E44BC18BA64F45AEAB2D8B:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:23,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:23,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E79C51E410E44BC18BA64F45AEAB2D8B being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:23,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:23,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:23,201 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E79C51E410E44BC18BA64F45AEAB2D8B: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:23,204 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E79C51E410E44BC18BA64F45AEAB2D8B: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:23,221 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E79C51E410E44BC18BA64F45AEAB2D8B with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:23,227 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E79C51E410E44BC18BA64F45AEAB2D8B
2016-05-18 00:57:23,227 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:23,228 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533041285_0001_1_01_000000_0
2016-05-18 00:57:23,232 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:23,239 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E79C51E410E44BC18BA64F45AEAB2D8B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:23,245 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 106531C925854ECBAE53C379BAEDF788
2016-05-18 00:57:23,246 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_-1"], id: 798B4E882BBE4B859073028CC8B85C38
2016-05-18 00:57:23,247 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_-1"]
2016-05-18 00:57:23,250 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 106531C925854ECBAE53C379BAEDF788, for 1 inputs
2016-05-18 00:57:23,252 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533041285_0001_1_00_000000_0_10003} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533043252, TimeTaken=5, Rate=0.03 MB/s
2016-05-18 00:57:23,252 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E79C51E410E44BC18BA64F45AEAB2D8B
2016-05-18 00:57:23,253 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:23,253 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E79C51E410E44BC18BA64F45AEAB2D8B, wasInterrupted=false
2016-05-18 00:57:23,254 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:23,254 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:23,255 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - E79C51E410E44BC18BA64F45AEAB2D8B: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:23,257 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:23,258 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001_wd/localmode-local-dir/output/attempt_1463533041285_0001_1_00_000000_0_10003/file.out, len=181, isLocal=true
2016-05-18 00:57:23,259 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:23,260 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:23,264 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:23,267 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:23,267 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E79C51E410E44BC18BA64F45AEAB2D8B
2016-05-18 00:57:23,268 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E79C51E410E44BC18BA64F45AEAB2D8B: Shuffle Runner thread complete
2016-05-18 00:57:23,268 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D5AB476D53F14ED293048EBC83CAF3FD, all 1 inputs ready in: 00:00:00.015
2016-05-18 00:57:23,270 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:23,271 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E79C51E410E44BC18BA64F45AEAB2D8B: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:23,293 INFO output.MROutput (MROutput.java:close(526)) - 798B4E882BBE4B859073028CC8B85C38 closed
2016-05-18 00:57:23,294 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533041285_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:23,296 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533041285_0001_1_01_000000_0
2016-05-18 00:57:23,296 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D5AB476D53F14ED293048EBC83CAF3FD, mem on close (mb), free: 69, total: 281, max: 672
2016-05-18 00:57:23,297 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E79C51E410E44BC18BA64F45AEAB2D8B
2016-05-18 00:57:23,297 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E79C51E410E44BC18BA64F45AEAB2D8B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:23,298 INFO output.MROutput (MROutput.java:close(526)) - 798B4E882BBE4B859073028CC8B85C38 closed
2016-05-18 00:57:23,298 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:23 Completed running task attempt: attempt_1463533041285_0001_1_01_000000_0
2016-05-18 00:57:23,300 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533041285_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:23,301 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533041285_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:23,303 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533041285_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=51, MERGE_PHASE_TIME=65, FIRST_EVENT_RECEIVED=37, LAST_EVENT_RECEIVED=37][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533043269, Process_Duration=27, Process_End_Time=1463533043296, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:23,305 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533041285_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:23,305 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D5AB476D53F14ED293048EBC83CAF3FD, taskAttemptId=attempt_1463533041285_0001_1_01_000000_0, creationTime=1463533043144, allocationTime=1463533043145, startTime=1463533043148, finishTime=1463533043305, timeTaken=157, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:23,308 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533041285_0001_1_01_000000_0
2016-05-18 00:57:23,306 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:23,311 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:23,311 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:23,311 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533041285_0001_00_000002
2016-05-18 00:57:23,311 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:TASK_FINISHED]: vertexName=D5AB476D53F14ED293048EBC83CAF3FD, taskId=task_1463533041285_0001_1_01_000000, startTime=1463533043148, finishTime=1463533043308, timeTaken=160, status=SUCCEEDED, successfulAttemptID=attempt_1463533041285_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=51, MERGE_PHASE_TIME=65, FIRST_EVENT_RECEIVED=37, LAST_EVENT_RECEIVED=37, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533043269, Process_Duration=27, Process_End_Time=1463533043296, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:23,312 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:23,312 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD]
2016-05-18 00:57:23,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:VERTEX_FINISHED]: vertexName=D5AB476D53F14ED293048EBC83CAF3FD, vertexId=vertex_1463533041285_0001_1_01, initRequestedTime=1463533042415, initedTime=1463533042493, startRequestedTime=1463533042561, startedTime=1463533042562, finishTime=1463533043312, timeTaken=750, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=51, MERGE_PHASE_TIME=65, FIRST_EVENT_RECEIVED=37, LAST_EVENT_RECEIVED=37, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533043269, Process_Duration=27, Process_End_Time=1463533043296, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533043148, firstTasksToStart=[ task_1463533041285_0001_1_01_000000 ], lastTaskFinishTime=1463533043305, lastTasksToFinish=[ task_1463533041285_0001_1_01_000000 ], minTaskDuration=157, maxTaskDuration=157, avgTaskDuration=157.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533041285_0001_1_01_000000 ], longestDurationTasks=[ task_1463533041285_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:23,315 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:23,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533041285_0001_00_000002, stoppedTime=1463533043315, exitStatus=0
2016-05-18 00:57:23,316 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:23,316 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:23,316 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533041285_0001_1_00 [E79C51E410E44BC18BA64F45AEAB2D8B]
2016-05-18 00:57:23,318 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533041285_0001_1
2016-05-18 00:57:23,319 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533041285_0001_1
2016-05-18 00:57:23,319 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533041285_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:23,319 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 798B4E882BBE4B859073028CC8B85C38 for vertex: vertex_1463533041285_0001_1_01 [D5AB476D53F14ED293048EBC83CAF3FD], outputName: 798B4E882BBE4B859073028CC8B85C38
2016-05-18 00:57:23,321 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533041285_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:23,323 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533041285_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:23,323 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533041285_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:23,323 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533041285_0001_00_000002, Exiting
2016-05-18 00:57:23,324 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533041285_0001_00_000002
2016-05-18 00:57:23,324 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533041285_0001_00_000002
2016-05-18 00:57:23,325 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533041285_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:23,325 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533041285_0001_00_000002 completed successfully
2016-05-18 00:57:23,333 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:798B4E882BBE4B859073028CC8B85C38 of vertex/vertexGroup:D5AB476D53F14ED293048EBC83CAF3FD isVertexGroupOutput:false
2016-05-18 00:57:23,333 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:23,340 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533041285_0001_1, queueSize=0
2016-05-18 00:57:23,342 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533041285_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533041285_0001_1, startTime=1463533042381, finishTime=1463533043333, timeTaken=952, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=2320, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=51, MERGE_PHASE_TIME=65, FIRST_EVENT_RECEIVED=37, LAST_EVENT_RECEIVED=37, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066086298, Process_Duration=38, Process_End_Time=2927066086336, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=4, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:23,344 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533041285_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:23,344 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533041285_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:23 Completed Dag: dag_1463533041285_0001_1
2016-05-18 00:57:23,346 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533041285_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:23,346 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533041285_0001_1_post
2016-05-18 00:57:23,347 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:23,347 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:23,348 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:23,349 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443] every/(1/1) ...blyplatform/-1_'field'_-1, with id=dag_1463533041285_0001_1
2016-05-18 00:57:23,352 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443] every/(1/1) ...blyplatform/-1_'field'_-1, with id=dag_1463533041285_0001_1
2016-05-18 00:57:23,354 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...blyplatform/-1_'field'_-1, applicationId=application_1463533041285_0001
2016-05-18 00:57:23,359 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:23,359 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:23,359 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:23,360 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:23,360 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:23,361 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:23,364 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:23,364 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:23,364 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:23,364 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:23,364 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:23,365 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:23,366 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:23,366 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:23,370 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5F96A081505F465C9E3EC72DF1D5B222/A4492071B89C4C6FA6EEF965AE889443/.tez/application_1463533041285_0001
2016-05-18 00:57:23,373 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_-1/_temporary
2016-05-18 00:57:23,373 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:02.302
2016-05-18 00:57:23,519 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_'field'_-1
2016-05-18 00:57:23,561 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:23,561 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:23,646 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.037
2016-05-18 00:57:23,664 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.056
2016-05-18 00:57:23,665 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:23,665 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:23,666 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:23,687 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:23,702 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_'field'
2016-05-18 00:57:23,779 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:23,779 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:23,779 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_'field'"]
2016-05-18 00:57:23,779 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:23,780 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:23,780 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:23,782 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...atform/-1_'field'_'field'
2016-05-18 00:57:23,782 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:23,807 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:23,812 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:23,813 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:23,813 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:23,824 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001 doesn't exist and is created
2016-05-18 00:57:23,848 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:23,848 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:23,850 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001_wd
2016-05-18 00:57:23,863 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533043813_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:23,863 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:23,863 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:23,865 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:23,865 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:23,866 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:23,866 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:23,867 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:23,874 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:23,874 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:23,874 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:23,874 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:23,874 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:23,874 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:23,874 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:23,875 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:23,875 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:23,875 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:23,875 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:23,875 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:23,881 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:23,882 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:23,883 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:23,883 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001_wd/localmode-log-dir
2016-05-18 00:57:23,884 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001_wd/localmode-log-dir/history.txt.appattempt_1463533043813_0001_000000, maxErrors=10
2016-05-18 00:57:23,884 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:23,884 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533043813_0001_000000, appSubmitTime=1463533043862, launchTime=1463533043862
2016-05-18 00:57:23,900 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:23,909 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:50918
2016-05-18 00:57:23,911 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533043813
2016-05-18 00:57:23,915 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533043813_0001_000000, startTime=1463533043915
2016-05-18 00:57:23,915 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:23,949 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:23,949 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:23,950 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...atform/-1_'field'_'field', applicationId=application_1463533043813_0001, dagName=[5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5] every/(1/1) ...atform/-1_'field'_'field'
2016-05-18 00:57:23,974 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5] every/(1/1) ...atform/-1_'field'_'field'
2016-05-18 00:57:23,974 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:23,976 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533043813_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001_wd/localmode-log-dir/dag_1463533043813_0001_1.dot
2016-05-18 00:57:23,977 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001_wd/localmode-log-dir/dag_1463533043813_0001_1-tez-dag.pb.txt
2016-05-18 00:57:23,985 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533043813_0001_1
2016-05-18 00:57:23,985 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:23,985 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5] every/(1/1) ...atform/-1_'field'_'field'
2016-05-18 00:57:23 Running Dag: dag_1463533043813_0001_1
2016-05-18 00:57:23,997 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533043813_0001_1, submitTime=1463533043974
2016-05-18 00:57:23,998 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001/tez-conf.pb" } size: 36913 timestamp: 1463533043000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:23,998 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:23,999 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:24,000 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001/tez-conf.pb" } size: 36913 timestamp: 1463533043000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:24,000 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]
2016-05-18 00:57:24,001 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:24,001 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:24,003 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533043813_0001_1, initTime=1463533044003
2016-05-18 00:57:24,003 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533043813_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:24,004 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:24,005 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:DAG_STARTED]: dagID=dag_1463533043813_0001_1, startTime=1463533044004
2016-05-18 00:57:24,005 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533043813_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:24,005 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: CDE85058C14247E298ABB73D21720644 : {93A3A6CCD3AB4FE1A4C6C8533B78B6F7={InputName=93A3A6CCD3AB4FE1A4C6C8533B78B6F7}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:24,005 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 93A3A6CCD3AB4FE1A4C6C8533B78B6F7, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:24,005 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,005 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,005 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,006 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,006 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:24,006 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]
2016-05-18 00:57:24,006 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 93A3A6CCD3AB4FE1A4C6C8533B78B6F7 on vertex vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,007 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...atform/-1_'field'_'field', applicationId=application_1463533043813_0001, dagName=[5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5] every/(1/1) ...atform/-1_'field'_'field'
2016-05-18 00:57:24,013 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:24,013 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]
2016-05-18 00:57:24,014 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]
2016-05-18 00:57:24,016 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533043813_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:24,016 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0], numAdditionalOutputs=1
2016-05-18 00:57:24,016 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=EACF00E29D824EBD9093A7405653D6BC, vertex=vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:24,017 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 93A3A6CCD3AB4FE1A4C6C8533B78B6F7 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:24,027 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for C4DF22D3E8DD44EBAE702DB930B852A0:EACF00E29D824EBD9093A7405653D6BC using oldmapred API
2016-05-18 00:57:24,028 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=EACF00E29D824EBD9093A7405653D6BC, vertexName=C4DF22D3E8DD44EBAE702DB930B852A0, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:24,030 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C4DF22D3E8DD44EBAE702DB930B852A0, vertexId=vertex_1463533043813_0001_1_01, initRequestedTime=1463533044006, initedTime=1463533044016, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:24,031 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:24,031 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:24,031 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:24,032 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:24,031 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:24,033 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 93A3A6CCD3AB4FE1A4C6C8533B78B6F7 on vertex vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,036 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:24,038 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {93A3A6CCD3AB4FE1A4C6C8533B78B6F7=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:24,040 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] parallelism set to 1
2016-05-18 00:57:24,040 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:24,040 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CDE85058C14247E298ABB73D21720644, vertexId=vertex_1463533043813_0001_1_00, initRequestedTime=1463533044005, initedTime=1463533044040, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:24,041 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] with distanceFromRoot: 0
2016-05-18 00:57:24,041 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:24,041 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533043813_0001_1_00, startRequestedTime=1463533044006, startedTime=1463533044041
2016-05-18 00:57:24,041 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533043813_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:24,041 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:24,042 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533043813_0001_1_00 for vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0] numStartedSources: 1 numSources: 1
2016-05-18 00:57:24,042 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0] with distanceFromRoot: 1
2016-05-18 00:57:24,042 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533043813_0001_1_01, startRequestedTime=1463533044042, startedTime=1463533044042
2016-05-18 00:57:24,045 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:24,045 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in CDE85058C14247E298ABB73D21720644
2016-05-18 00:57:24,046 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: C4DF22D3E8DD44EBAE702DB930B852A0 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:24,047 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,049 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: CDE85058C14247E298ABB73D21720644 in vertex: C4DF22D3E8DD44EBAE702DB930B852A0 numBipartiteSourceTasks: 1
2016-05-18 00:57:24,049 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:TASK_STARTED]: vertexName=CDE85058C14247E298ABB73D21720644, taskId=task_1463533043813_0001_1_00_000000, scheduledTime=1463533044049, launchTime=1463533044049
2016-05-18 00:57:24,050 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,051 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:24,054 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533043813_0001_00_000001, launchTime=1463533044054
2016-05-18 00:57:24,055 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533043813_0001_00_000001
2016-05-18 00:57:24,056 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533043813_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533043813_0001_1_00_000000_0
2016-05-18 00:57:24,056 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533043813_0001_1_00_000000_0] started. Is using containerId: [container_1463533043813_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:24,057 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CDE85058C14247E298ABB73D21720644, taskAttemptId=attempt_1463533043813_0001_1_00_000000_0, startTime=1463533044056, containerId=container_1463533043813_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:24 Starting to run new task attempt: attempt_1463533043813_0001_1_00_000000_0
2016-05-18 00:57:24,058 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533043813_0001_1_00_000000_0
2016-05-18 00:57:24,058 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:24,058 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:24,060 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533043813_0001, with dag id: dag_1463533043813_0001_1
2016-05-18 00:57:24,062 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5] every/(1/1) ...atform/-1_'field'_'field', VertexName: CDE85058C14247E298ABB73D21720644, VertexParallelism: 1, TaskAttemptID:attempt_1463533043813_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=93A3A6CCD3AB4FE1A4C6C8533B78B6F7, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C4DF22D3E8DD44EBAE702DB930B852A0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:24,062 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:24,062 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533043813_0001_1_00_000000_0
2016-05-18 00:57:24,065 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533043813_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:24,066 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533043813_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:24,077 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:24,078 INFO input.MRInput (MRInput.java:initialize(440)) - 93A3A6CCD3AB4FE1A4C6C8533B78B6F7 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:24,079 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 93A3A6CCD3AB4FE1A4C6C8533B78B6F7
2016-05-18 00:57:24,086 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CDE85058C14247E298ABB73D21720644, ordinal: 0
2016-05-18 00:57:24,086 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CDE85058C14247E298ABB73D21720644, mem on start (mb), free: 169, total: 397, max: 672
2016-05-18 00:57:24,087 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:24,087 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:24,087 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:24,088 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:24,088 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[93A3A6CCD3AB4FE1A4C6C8533B78B6F7:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [C4DF22D3E8DD44EBAE702DB930B852A0:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:24,089 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:24,089 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:24,089 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:24,090 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[93A3A6CCD3AB4FE1A4C6C8533B78B6F7:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [C4DF22D3E8DD44EBAE702DB930B852A0:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:24,090 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:24,091 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 93A3A6CCD3AB4FE1A4C6C8533B78B6F7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:24,091 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:24,092 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:24,092 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 93A3A6CCD3AB4FE1A4C6C8533B78B6F7
2016-05-18 00:57:24,093 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:24,093 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533043813_0001_1_00_000000_0
2016-05-18 00:57:24,095 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:24,095 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 93A3A6CCD3AB4FE1A4C6C8533B78B6F7 initialized RecordReader from event
2016-05-18 00:57:24,099 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:24,100 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 93A3A6CCD3AB4FE1A4C6C8533B78B6F7
2016-05-18 00:57:24,100 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 749D4FF89A0C4B15BA3A7B8958574D09
2016-05-18 00:57:24,100 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 749D4FF89A0C4B15BA3A7B8958574D09
2016-05-18 00:57:24,103 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:24,106 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:24,106 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - C4DF22D3E8DD44EBAE702DB930B852A0 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@79e0b66f, comparator=cascading.tuple.hadoop.util.TupleComparator@26b2f71d, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:24,107 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for C4DF22D3E8DD44EBAE702DB930B852A0: , UsingHashComparator=false
2016-05-18 00:57:24,194 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:24,194 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:24,195 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for C4DF22D3E8DD44EBAE702DB930B852A0: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:24,195 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - C4DF22D3E8DD44EBAE702DB930B852A0: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:24,196 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:24,196 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CDE85058C14247E298ABB73D21720644, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:24,200 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533043813_0001_1_00_000000_0
2016-05-18 00:57:24,200 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CDE85058C14247E298ABB73D21720644, mem on close (mb), free: 66, total: 397, max: 672
2016-05-18 00:57:24,201 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - C4DF22D3E8DD44EBAE702DB930B852A0: Starting flush of map output
2016-05-18 00:57:24,202 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - C4DF22D3E8DD44EBAE702DB930B852A0: Span0.length = 20, perItem = 6
2016-05-18 00:57:24,202 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - C4DF22D3E8DD44EBAE702DB930B852A0: done sorting span=0, length=20, time=0
2016-05-18 00:57:24,203 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - C4DF22D3E8DD44EBAE702DB930B852A0: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:24,205 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - C4DF22D3E8DD44EBAE702DB930B852A0: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001_wd/localmode-local-dir/output/attempt_1463533043813_0001_1_00_000000_0_10009_0/file.out
2016-05-18 00:57:24,210 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:24 Completed running task attempt: attempt_1463533043813_0001_1_00_000000_0
2016-05-18 00:57:24,211 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533043813_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:24,212 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533043813_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:24,213 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533043813_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533044197, Process_Duration=3, Process_End_Time=1463533044200, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:24,215 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533043813_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:24,215 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CDE85058C14247E298ABB73D21720644, taskAttemptId=attempt_1463533043813_0001_1_00_000000_0, creationTime=1463533044050, allocationTime=1463533044053, startTime=1463533044056, finishTime=1463533044215, timeTaken=159, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:24,215 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:24,216 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:24,216 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533043813_0001_1_00_000000_0
2016-05-18 00:57:24,217 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:24,218 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533043813_0001_00_000001
2016-05-18 00:57:24,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:TASK_FINISHED]: vertexName=CDE85058C14247E298ABB73D21720644, taskId=task_1463533043813_0001_1_00_000000, startTime=1463533044056, finishTime=1463533044217, timeTaken=161, status=SUCCEEDED, successfulAttemptID=attempt_1463533043813_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533044197, Process_Duration=3, Process_End_Time=1463533044200, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:24,220 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:24,220 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,221 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:VERTEX_FINISHED]: vertexName=CDE85058C14247E298ABB73D21720644, vertexId=vertex_1463533043813_0001_1_00, initRequestedTime=1463533044005, initedTime=1463533044040, startRequestedTime=1463533044006, startedTime=1463533044041, finishTime=1463533044220, timeTaken=179, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533044197, Process_Duration=3, Process_End_Time=1463533044200, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533044056, firstTasksToStart=[ task_1463533043813_0001_1_00_000000 ], lastTaskFinishTime=1463533044215, lastTasksToFinish=[ task_1463533043813_0001_1_00_000000 ], minTaskDuration=159, maxTaskDuration=159, avgTaskDuration=159.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533043813_0001_1_00_000000 ], longestDurationTasks=[ task_1463533043813_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:24,221 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:24,222 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533043813_0001_00_000001, stoppedTime=1463533044222, exitStatus=0
2016-05-18 00:57:24,222 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0] attempt: attempt_1463533043813_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:24,224 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C4DF22D3E8DD44EBAE702DB930B852A0
2016-05-18 00:57:24,225 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=CDE85058C14247E298ABB73D21720644, destinationVertexName=C4DF22D3E8DD44EBAE702DB930B852A0. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:24,225 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]
2016-05-18 00:57:24,224 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:24,225 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:24,226 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533043813_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:24,227 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:TASK_STARTED]: vertexName=C4DF22D3E8DD44EBAE702DB930B852A0, taskId=task_1463533043813_0001_1_01_000000, scheduledTime=1463533044226, launchTime=1463533044226
2016-05-18 00:57:24,227 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]
2016-05-18 00:57:24,229 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533043813_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:24,229 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533043813_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:24,229 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533043813_0001_00_000001, Exiting
2016-05-18 00:57:24,230 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533043813_0001_00_000001
2016-05-18 00:57:24,230 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533043813_0001_00_000001
2016-05-18 00:57:24,230 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533043813_0001_00_000002, launchTime=1463533044229
2016-05-18 00:57:24,231 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533043813_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:24,231 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533043813_0001_00_000001 completed successfully
2016-05-18 00:57:24,232 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533043813_0001_00_000002
2016-05-18 00:57:24,232 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533043813_0001_1_01_000000_0] started. Is using containerId: [container_1463533043813_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:24,233 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533043813_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533043813_0001_1_01_000000_0
2016-05-18 00:57:24,233 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C4DF22D3E8DD44EBAE702DB930B852A0, taskAttemptId=attempt_1463533043813_0001_1_01_000000_0, startTime=1463533044232, containerId=container_1463533043813_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:24 Starting to run new task attempt: attempt_1463533043813_0001_1_01_000000_0
2016-05-18 00:57:24,234 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533043813_0001_1_01_000000_0
2016-05-18 00:57:24,234 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:24,234 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:24,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5] every/(1/1) ...atform/-1_'field'_'field', VertexName: C4DF22D3E8DD44EBAE702DB930B852A0, VertexParallelism: 1, TaskAttemptID:attempt_1463533043813_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CDE85058C14247E298ABB73D21720644, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EACF00E29D824EBD9093A7405653D6BC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:24,235 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:24,235 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533043813_0001_1_01_000000_0
2016-05-18 00:57:24,236 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533043813_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]
2016-05-18 00:57:24,237 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533043813_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:24,246 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:24,255 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:24,265 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C4DF22D3E8DD44EBAE702DB930B852A0, ordinal: 1
2016-05-18 00:57:24,265 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C4DF22D3E8DD44EBAE702DB930B852A0, mem on start (mb), free: 57, total: 397, max: 672
2016-05-18 00:57:24,265 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:24,265 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:24,266 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:24,271 INFO output.MROutput (MROutput.java:initialize(421)) - EACF00E29D824EBD9093A7405653D6BC: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:24,272 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:24,272 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[EACF00E29D824EBD9093A7405653D6BC:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [CDE85058C14247E298ABB73D21720644:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:24,272 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:24,272 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:24,272 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:24,273 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[EACF00E29D824EBD9093A7405653D6BC:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [CDE85058C14247E298ABB73D21720644:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:24,273 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:24,273 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CDE85058C14247E298ABB73D21720644 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:24,274 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - CDE85058C14247E298ABB73D21720644: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:24,274 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:24,274 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:24,274 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - CDE85058C14247E298ABB73D21720644: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:24,276 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: CDE85058C14247E298ABB73D21720644 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:24,277 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CDE85058C14247E298ABB73D21720644
2016-05-18 00:57:24,278 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:24,279 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533043813_0001_1_01_000000_0
2016-05-18 00:57:24,283 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CDE85058C14247E298ABB73D21720644: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:24,286 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533043813_0001_1_00_000000_0_10009} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533044286, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:24,286 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:24,286 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : CDE85058C14247E298ABB73D21720644
2016-05-18 00:57:24,286 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:24,287 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 749D4FF89A0C4B15BA3A7B8958574D09
2016-05-18 00:57:24,287 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_'field'"], id: EACF00E29D824EBD9093A7405653D6BC
2016-05-18 00:57:24,288 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: CDE85058C14247E298ABB73D21720644, wasInterrupted=false
2016-05-18 00:57:24,288 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_'field'"]
2016-05-18 00:57:24,288 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:24,289 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 749D4FF89A0C4B15BA3A7B8958574D09, for 1 inputs
2016-05-18 00:57:24,290 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:24,293 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:24,293 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001_wd/localmode-local-dir/output/attempt_1463533043813_0001_1_00_000000_0_10009/file.out, len=181, isLocal=true
2016-05-18 00:57:24,293 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:24,293 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:24,294 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:24,295 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:24,296 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : CDE85058C14247E298ABB73D21720644
2016-05-18 00:57:24,296 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C4DF22D3E8DD44EBAE702DB930B852A0, all 1 inputs ready in: 00:00:00.006
2016-05-18 00:57:24,297 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - CDE85058C14247E298ABB73D21720644: Shuffle Runner thread complete
2016-05-18 00:57:24,298 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:24,298 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - CDE85058C14247E298ABB73D21720644: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:24,305 INFO output.MROutput (MROutput.java:close(526)) - EACF00E29D824EBD9093A7405653D6BC closed
2016-05-18 00:57:24,306 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533043813_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:24,307 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533043813_0001_1_01_000000_0
2016-05-18 00:57:24,307 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C4DF22D3E8DD44EBAE702DB930B852A0, mem on close (mb), free: 51, total: 397, max: 672
2016-05-18 00:57:24,307 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: CDE85058C14247E298ABB73D21720644
2016-05-18 00:57:24,307 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CDE85058C14247E298ABB73D21720644: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:24,308 INFO output.MROutput (MROutput.java:close(526)) - EACF00E29D824EBD9093A7405653D6BC closed
2016-05-18 00:57:24,308 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:24 Completed running task attempt: attempt_1463533043813_0001_1_01_000000_0
2016-05-18 00:57:24,309 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533043813_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:24,310 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533043813_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:24,310 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533043813_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=21, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533044297, Process_Duration=10, Process_End_Time=1463533044307, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:24,315 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533043813_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:24,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C4DF22D3E8DD44EBAE702DB930B852A0, taskAttemptId=attempt_1463533043813_0001_1_01_000000_0, creationTime=1463533044227, allocationTime=1463533044228, startTime=1463533044232, finishTime=1463533044315, timeTaken=83, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:24,315 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:24,317 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:24,316 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533043813_0001_1_01_000000_0
2016-05-18 00:57:24,317 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:24,318 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533043813_0001_00_000002
2016-05-18 00:57:24,319 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:TASK_FINISHED]: vertexName=C4DF22D3E8DD44EBAE702DB930B852A0, taskId=task_1463533043813_0001_1_01_000000, startTime=1463533044232, finishTime=1463533044317, timeTaken=85, status=SUCCEEDED, successfulAttemptID=attempt_1463533043813_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=21, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533044297, Process_Duration=10, Process_End_Time=1463533044307, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:24,319 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:24,319 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0]
2016-05-18 00:57:24,322 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:VERTEX_FINISHED]: vertexName=C4DF22D3E8DD44EBAE702DB930B852A0, vertexId=vertex_1463533043813_0001_1_01, initRequestedTime=1463533044006, initedTime=1463533044016, startRequestedTime=1463533044042, startedTime=1463533044042, finishTime=1463533044320, timeTaken=278, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=21, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533044297, Process_Duration=10, Process_End_Time=1463533044307, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533044232, firstTasksToStart=[ task_1463533043813_0001_1_01_000000 ], lastTaskFinishTime=1463533044315, lastTasksToFinish=[ task_1463533043813_0001_1_01_000000 ], minTaskDuration=83, maxTaskDuration=83, avgTaskDuration=83.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533043813_0001_1_01_000000 ], longestDurationTasks=[ task_1463533043813_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:24,322 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:24,323 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533043813_0001_00_000002, stoppedTime=1463533044322, exitStatus=0
2016-05-18 00:57:24,326 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:24,326 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:24,327 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533043813_0001_1_00 [CDE85058C14247E298ABB73D21720644]
2016-05-18 00:57:24,327 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533043813_0001_1
2016-05-18 00:57:24,328 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533043813_0001_1
2016-05-18 00:57:24,328 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: EACF00E29D824EBD9093A7405653D6BC for vertex: vertex_1463533043813_0001_1_01 [C4DF22D3E8DD44EBAE702DB930B852A0], outputName: EACF00E29D824EBD9093A7405653D6BC
2016-05-18 00:57:24,329 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533043813_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:24,329 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533043813_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:24,329 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533043813_0001_00_000002, Exiting
2016-05-18 00:57:24,333 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533043813_0001_00_000002
2016-05-18 00:57:24,333 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533043813_0001_00_000002
2016-05-18 00:57:24,329 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533043813_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:24,335 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533043813_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:24,336 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533043813_0001_00_000002 completed successfully
2016-05-18 00:57:24,336 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533043813_0001_00_000002
2016-05-18 00:57:24,342 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:EACF00E29D824EBD9093A7405653D6BC of vertex/vertexGroup:C4DF22D3E8DD44EBAE702DB930B852A0 isVertexGroupOutput:false
2016-05-18 00:57:24,344 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:24,349 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533043813_0001_1, queueSize=0
2016-05-18 00:57:24,351 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533043813_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533043813_0001_1, startTime=1463533044004, finishTime=1463533044345, timeTaken=341, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=620, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=14, MERGE_PHASE_TIME=21, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066088494, Process_Duration=13, Process_End_Time=2927066088507, Read_Duration=5, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:24,351 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533043813_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:24,351 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533043813_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:24 Completed Dag: dag_1463533043813_0001_1
2016-05-18 00:57:24,354 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533043813_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:24,355 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533043813_0001_1_post
2016-05-18 00:57:24,355 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:24,356 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:24,357 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:24,358 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5] every/(1/1) ...atform/-1_'field'_'field', with id=dag_1463533043813_0001_1
2016-05-18 00:57:24,358 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5] every/(1/1) ...atform/-1_'field'_'field', with id=dag_1463533043813_0001_1
2016-05-18 00:57:24,361 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...atform/-1_'field'_'field', applicationId=application_1463533043813_0001
2016-05-18 00:57:24,365 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:24,365 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:24,365 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:24,366 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:24,366 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:24,366 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:24,367 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:24,367 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:24,367 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:24,368 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:24,368 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:24,368 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:24,369 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:24,371 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:24,374 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5991F788C86A4BA7B6FBB8B94BC536B4/A4686F83FEDC42D2B2E78A222C1418D5/.tez/application_1463533043813_0001
2016-05-18 00:57:24,375 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_'field'/_temporary
2016-05-18 00:57:24,376 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.597
2016-05-18 00:57:24,500 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_'field'_'field'
2016-05-18 00:57:24,540 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:24,540 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:24,608 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.030
2016-05-18 00:57:24,627 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.049
2016-05-18 00:57:24,628 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:24,628 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:24,629 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:24,648 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:24,657 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_RESULTS
2016-05-18 00:57:24,729 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:24,729 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:24,730 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_RESULTS"]
2016-05-18 00:57:24,730 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:24,730 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:24,730 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:24,731 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...atform/-1_'field'_RESULTS
2016-05-18 00:57:24,731 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:24,747 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:24,752 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:24,752 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:24,753 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:24,762 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001 doesn't exist and is created
2016-05-18 00:57:24,783 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:24,783 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:24,784 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001_wd
2016-05-18 00:57:24,798 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533044752_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:24,798 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:24,798 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:24,800 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:24,801 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:24,801 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:24,801 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:24,802 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:24,804 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:24,804 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:24,804 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:24,805 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:24,805 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:24,805 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:24,806 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:24,806 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:24,806 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:24,806 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:24,807 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:24,807 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:24,809 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:24,810 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:24,810 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:24,811 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001_wd/localmode-log-dir
2016-05-18 00:57:24,811 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001_wd/localmode-log-dir/history.txt.appattempt_1463533044752_0001_000000, maxErrors=10
2016-05-18 00:57:24,812 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:24,812 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533044752_0001_000000, appSubmitTime=1463533044797, launchTime=1463533044797
2016-05-18 00:57:24,821 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:24,826 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:56044
2016-05-18 00:57:24,826 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533044752
2016-05-18 00:57:24,829 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533044752_0001_000000, startTime=1463533044828
2016-05-18 00:57:24,829 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:24,884 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:24,884 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:24,884 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...atform/-1_'field'_RESULTS, applicationId=application_1463533044752_0001, dagName=[DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688] every/(1/1) ...atform/-1_'field'_RESULTS
2016-05-18 00:57:24,905 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688] every/(1/1) ...atform/-1_'field'_RESULTS
2016-05-18 00:57:24,905 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:24,906 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533044752_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001_wd/localmode-log-dir/dag_1463533044752_0001_1.dot
2016-05-18 00:57:24,907 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001_wd/localmode-log-dir/dag_1463533044752_0001_1-tez-dag.pb.txt
2016-05-18 00:57:24,915 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533044752_0001_1
2016-05-18 00:57:24,915 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:24,916 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688] every/(1/1) ...atform/-1_'field'_RESULTS
2016-05-18 00:57:24 Running Dag: dag_1463533044752_0001_1
2016-05-18 00:57:24,926 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533044752_0001_1, submitTime=1463533044905
2016-05-18 00:57:24,928 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001/tez-conf.pb" } size: 36913 timestamp: 1463533044000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:24,928 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,928 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:24,929 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001/tez-conf.pb" } size: 36913 timestamp: 1463533044000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:24,929 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]
2016-05-18 00:57:24,930 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:24,930 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:24,931 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533044752_0001_1, initTime=1463533044930
2016-05-18 00:57:24,931 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533044752_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:24,932 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:24,934 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:DAG_STARTED]: dagID=dag_1463533044752_0001_1, startTime=1463533044934
2016-05-18 00:57:24,934 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533044752_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:24,934 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F8CB379FAE8A4DDF88CE5AA880A9C26D : {4DF23ECA4BAB4F0C95794C62A72FD34F={InputName=4DF23ECA4BAB4F0C95794C62A72FD34F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:24,934 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 4DF23ECA4BAB4F0C95794C62A72FD34F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:24,934 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,934 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,935 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,935 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,935 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...atform/-1_'field'_RESULTS, applicationId=application_1463533044752_0001, dagName=[DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688] every/(1/1) ...atform/-1_'field'_RESULTS
2016-05-18 00:57:24,935 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:24,936 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 4DF23ECA4BAB4F0C95794C62A72FD34F on vertex vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,937 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]
2016-05-18 00:57:24,938 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 4DF23ECA4BAB4F0C95794C62A72FD34F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:24,941 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:24,941 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]
2016-05-18 00:57:24,942 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]
2016-05-18 00:57:24,942 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:24,943 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533044752_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:24,944 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD], numAdditionalOutputs=1
2016-05-18 00:57:24,945 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=968C99D1C0434F21ABB097616EE732E5, vertex=vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:24,944 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:24,946 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:24,946 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:24,947 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 4DF23ECA4BAB4F0C95794C62A72FD34F on vertex vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,948 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for AF72E618E89E4F238EA3F250EF1AADDD:968C99D1C0434F21ABB097616EE732E5 using oldmapred API
2016-05-18 00:57:24,948 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=968C99D1C0434F21ABB097616EE732E5, vertexName=AF72E618E89E4F238EA3F250EF1AADDD, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:24,950 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AF72E618E89E4F238EA3F250EF1AADDD, vertexId=vertex_1463533044752_0001_1_01, initRequestedTime=1463533044937, initedTime=1463533044944, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:24,950 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:24,951 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:24,952 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {4DF23ECA4BAB4F0C95794C62A72FD34F=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:24,953 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] parallelism set to 1
2016-05-18 00:57:24,953 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:24,954 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F8CB379FAE8A4DDF88CE5AA880A9C26D, vertexId=vertex_1463533044752_0001_1_00, initRequestedTime=1463533044934, initedTime=1463533044954, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:24,954 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] with distanceFromRoot: 0
2016-05-18 00:57:24,954 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:24,955 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533044752_0001_1_00, startRequestedTime=1463533044936, startedTime=1463533044954
2016-05-18 00:57:24,956 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533044752_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:24,956 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:24,956 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533044752_0001_1_00 for vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD] numStartedSources: 1 numSources: 1
2016-05-18 00:57:24,956 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD] with distanceFromRoot: 1
2016-05-18 00:57:24,957 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533044752_0001_1_01, startRequestedTime=1463533044956, startedTime=1463533044956
2016-05-18 00:57:24,957 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:24,958 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: AF72E618E89E4F238EA3F250EF1AADDD with 0 source tasks and 1 pending tasks
2016-05-18 00:57:24,958 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F8CB379FAE8A4DDF88CE5AA880A9C26D
2016-05-18 00:57:24,961 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,961 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:TASK_STARTED]: vertexName=F8CB379FAE8A4DDF88CE5AA880A9C26D, taskId=task_1463533044752_0001_1_00_000000, scheduledTime=1463533044961, launchTime=1463533044961
2016-05-18 00:57:24,962 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:24,962 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F8CB379FAE8A4DDF88CE5AA880A9C26D in vertex: AF72E618E89E4F238EA3F250EF1AADDD numBipartiteSourceTasks: 1
2016-05-18 00:57:24,963 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:24,966 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533044752_0001_00_000001, launchTime=1463533044966
2016-05-18 00:57:24,967 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533044752_0001_00_000001
2016-05-18 00:57:24,967 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533044752_0001, with dag id: dag_1463533044752_0001_1
2016-05-18 00:57:24,968 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533044752_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533044752_0001_1_00_000000_0
2016-05-18 00:57:24,968 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533044752_0001_1_00_000000_0] started. Is using containerId: [container_1463533044752_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:24,969 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F8CB379FAE8A4DDF88CE5AA880A9C26D, taskAttemptId=attempt_1463533044752_0001_1_00_000000_0, startTime=1463533044967, containerId=container_1463533044752_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:24 Starting to run new task attempt: attempt_1463533044752_0001_1_00_000000_0
2016-05-18 00:57:24,972 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533044752_0001_1_00_000000_0
2016-05-18 00:57:24,974 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:24,975 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:24,975 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688] every/(1/1) ...atform/-1_'field'_RESULTS, VertexName: F8CB379FAE8A4DDF88CE5AA880A9C26D, VertexParallelism: 1, TaskAttemptID:attempt_1463533044752_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=4DF23ECA4BAB4F0C95794C62A72FD34F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=AF72E618E89E4F238EA3F250EF1AADDD, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:24,976 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:24,976 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533044752_0001_1_00_000000_0
2016-05-18 00:57:24,979 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533044752_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:24,980 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533044752_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:24,984 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:24,985 INFO input.MRInput (MRInput.java:initialize(440)) - 4DF23ECA4BAB4F0C95794C62A72FD34F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:24,985 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 4DF23ECA4BAB4F0C95794C62A72FD34F
2016-05-18 00:57:24,994 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F8CB379FAE8A4DDF88CE5AA880A9C26D, ordinal: 0
2016-05-18 00:57:24,994 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F8CB379FAE8A4DDF88CE5AA880A9C26D, mem on start (mb), free: 163, total: 397, max: 672
2016-05-18 00:57:24,995 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:24,995 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:24,996 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:24,996 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:24,996 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[4DF23ECA4BAB4F0C95794C62A72FD34F:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [AF72E618E89E4F238EA3F250EF1AADDD:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:24,997 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:24,997 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:24,998 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:24,998 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[4DF23ECA4BAB4F0C95794C62A72FD34F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [AF72E618E89E4F238EA3F250EF1AADDD:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:24,999 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:24,999 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 4DF23ECA4BAB4F0C95794C62A72FD34F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:24,999 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 4DF23ECA4BAB4F0C95794C62A72FD34F
2016-05-18 00:57:25,000 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:25,001 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:25,001 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:25,002 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533044752_0001_1_00_000000_0
2016-05-18 00:57:25,004 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:25,004 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 4DF23ECA4BAB4F0C95794C62A72FD34F initialized RecordReader from event
2016-05-18 00:57:25,006 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:25,008 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 4DF23ECA4BAB4F0C95794C62A72FD34F
2016-05-18 00:57:25,008 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: C544969E82FA400694A3A87C959831EE
2016-05-18 00:57:25,008 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] C544969E82FA400694A3A87C959831EE
2016-05-18 00:57:25,010 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:25,014 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:25,015 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - AF72E618E89E4F238EA3F250EF1AADDD using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@16cf2484, comparator=cascading.tuple.hadoop.util.TupleComparator@796661ce, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:25,016 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for AF72E618E89E4F238EA3F250EF1AADDD: , UsingHashComparator=false
2016-05-18 00:57:25,030 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:25,030 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:25,031 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for AF72E618E89E4F238EA3F250EF1AADDD: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:25,032 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - AF72E618E89E4F238EA3F250EF1AADDD: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:25,032 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:25,033 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F8CB379FAE8A4DDF88CE5AA880A9C26D, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:25,037 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533044752_0001_1_00_000000_0
2016-05-18 00:57:25,037 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F8CB379FAE8A4DDF88CE5AA880A9C26D, mem on close (mb), free: 60, total: 397, max: 672
2016-05-18 00:57:25,038 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - AF72E618E89E4F238EA3F250EF1AADDD: Starting flush of map output
2016-05-18 00:57:25,039 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - AF72E618E89E4F238EA3F250EF1AADDD: Span0.length = 20, perItem = 6
2016-05-18 00:57:25,039 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - AF72E618E89E4F238EA3F250EF1AADDD: done sorting span=0, length=20, time=0
2016-05-18 00:57:25,040 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - AF72E618E89E4F238EA3F250EF1AADDD: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:25,042 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - AF72E618E89E4F238EA3F250EF1AADDD: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001_wd/localmode-local-dir/output/attempt_1463533044752_0001_1_00_000000_0_10015_0/file.out
2016-05-18 00:57:25,049 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:25 Completed running task attempt: attempt_1463533044752_0001_1_00_000000_0
2016-05-18 00:57:25,050 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533044752_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:25,051 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533044752_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:25,052 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533044752_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533045033, Process_Duration=4, Process_End_Time=1463533045037, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:25,053 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533044752_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:25,054 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:25,054 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F8CB379FAE8A4DDF88CE5AA880A9C26D, taskAttemptId=attempt_1463533044752_0001_1_00_000000_0, creationTime=1463533044962, allocationTime=1463533044965, startTime=1463533044967, finishTime=1463533045053, timeTaken=86, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:25,054 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:25,054 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533044752_0001_1_00_000000_0
2016-05-18 00:57:25,055 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,055 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533044752_0001_00_000001
2016-05-18 00:57:25,055 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:TASK_FINISHED]: vertexName=F8CB379FAE8A4DDF88CE5AA880A9C26D, taskId=task_1463533044752_0001_1_00_000000, startTime=1463533044967, finishTime=1463533045055, timeTaken=88, status=SUCCEEDED, successfulAttemptID=attempt_1463533044752_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533045033, Process_Duration=4, Process_End_Time=1463533045037, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:25,056 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:25,057 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:25,059 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:VERTEX_FINISHED]: vertexName=F8CB379FAE8A4DDF88CE5AA880A9C26D, vertexId=vertex_1463533044752_0001_1_00, initRequestedTime=1463533044934, initedTime=1463533044954, startRequestedTime=1463533044936, startedTime=1463533044954, finishTime=1463533045057, timeTaken=103, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533045033, Process_Duration=4, Process_End_Time=1463533045037, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533044967, firstTasksToStart=[ task_1463533044752_0001_1_00_000000 ], lastTaskFinishTime=1463533045053, lastTasksToFinish=[ task_1463533044752_0001_1_00_000000 ], minTaskDuration=86, maxTaskDuration=86, avgTaskDuration=86.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533044752_0001_1_00_000000 ], longestDurationTasks=[ task_1463533044752_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:25,059 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:25,059 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533044752_0001_00_000001, stoppedTime=1463533045059, exitStatus=0
2016-05-18 00:57:25,060 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD] attempt: attempt_1463533044752_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:25,060 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AF72E618E89E4F238EA3F250EF1AADDD
2016-05-18 00:57:25,060 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F8CB379FAE8A4DDF88CE5AA880A9C26D, destinationVertexName=AF72E618E89E4F238EA3F250EF1AADDD. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:25,061 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]
2016-05-18 00:57:25,062 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:25,063 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:25,063 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533044752_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:25,064 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:TASK_STARTED]: vertexName=AF72E618E89E4F238EA3F250EF1AADDD, taskId=task_1463533044752_0001_1_01_000000, scheduledTime=1463533045064, launchTime=1463533045064
2016-05-18 00:57:25,065 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]
2016-05-18 00:57:25,066 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533044752_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:25,070 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533044752_0001_00_000001: 14 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:25,070 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533044752_0001_00_000001, Exiting
2016-05-18 00:57:25,071 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533044752_0001_00_000001
2016-05-18 00:57:25,071 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533044752_0001_00_000001
2016-05-18 00:57:25,073 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533044752_0001_00_000002, launchTime=1463533045072
2016-05-18 00:57:25,073 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533044752_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:25,074 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533044752_0001_00_000001 completed successfully
2016-05-18 00:57:25,074 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533044752_0001_00_000002
2016-05-18 00:57:25,075 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533044752_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533044752_0001_1_01_000000_0
2016-05-18 00:57:25,075 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533044752_0001_1_01_000000_0] started. Is using containerId: [container_1463533044752_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:25,076 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AF72E618E89E4F238EA3F250EF1AADDD, taskAttemptId=attempt_1463533044752_0001_1_01_000000_0, startTime=1463533045075, containerId=container_1463533044752_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:25 Starting to run new task attempt: attempt_1463533044752_0001_1_01_000000_0
2016-05-18 00:57:25,076 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533044752_0001_1_01_000000_0
2016-05-18 00:57:25,076 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,076 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:25,077 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688] every/(1/1) ...atform/-1_'field'_RESULTS, VertexName: AF72E618E89E4F238EA3F250EF1AADDD, VertexParallelism: 1, TaskAttemptID:attempt_1463533044752_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F8CB379FAE8A4DDF88CE5AA880A9C26D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=968C99D1C0434F21ABB097616EE732E5, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:25,077 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:25,077 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533044752_0001_1_01_000000_0
2016-05-18 00:57:25,079 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533044752_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]
2016-05-18 00:57:25,082 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533044752_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:25,084 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:25,086 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:25,092 INFO output.MROutput (MROutput.java:initialize(421)) - 968C99D1C0434F21ABB097616EE732E5: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:25,098 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AF72E618E89E4F238EA3F250EF1AADDD, ordinal: 1
2016-05-18 00:57:25,098 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AF72E618E89E4F238EA3F250EF1AADDD, mem on start (mb), free: 54, total: 397, max: 672
2016-05-18 00:57:25,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:25,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:25,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:25,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:25,098 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[968C99D1C0434F21ABB097616EE732E5:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [F8CB379FAE8A4DDF88CE5AA880A9C26D:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:25,099 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:25,099 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:25,099 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:25,099 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[968C99D1C0434F21ABB097616EE732E5:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [F8CB379FAE8A4DDF88CE5AA880A9C26D:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:25,099 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:25,100 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F8CB379FAE8A4DDF88CE5AA880A9C26D being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:25,100 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:25,100 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:25,100 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F8CB379FAE8A4DDF88CE5AA880A9C26D: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:25,101 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F8CB379FAE8A4DDF88CE5AA880A9C26D: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:25,102 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F8CB379FAE8A4DDF88CE5AA880A9C26D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:25,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F8CB379FAE8A4DDF88CE5AA880A9C26D
2016-05-18 00:57:25,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:25,104 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533044752_0001_1_01_000000_0
2016-05-18 00:57:25,105 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F8CB379FAE8A4DDF88CE5AA880A9C26D: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:25,108 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533044752_0001_1_00_000000_0_10015} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533045108, TimeTaken=3, Rate=0.06 MB/s
2016-05-18 00:57:25,108 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F8CB379FAE8A4DDF88CE5AA880A9C26D
2016-05-18 00:57:25,109 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:25,109 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F8CB379FAE8A4DDF88CE5AA880A9C26D, wasInterrupted=false
2016-05-18 00:57:25,110 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - F8CB379FAE8A4DDF88CE5AA880A9C26D: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:25,110 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:25,110 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:25,110 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:25,112 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:25,112 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001_wd/localmode-local-dir/output/attempt_1463533044752_0001_1_00_000000_0_10015/file.out, len=181, isLocal=true
2016-05-18 00:57:25,112 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:25,112 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:25,112 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:25,113 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:25,113 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F8CB379FAE8A4DDF88CE5AA880A9C26D
2016-05-18 00:57:25,113 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F8CB379FAE8A4DDF88CE5AA880A9C26D: Shuffle Runner thread complete
2016-05-18 00:57:25,114 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: C544969E82FA400694A3A87C959831EE
2016-05-18 00:57:25,115 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_RESULTS"], id: 968C99D1C0434F21ABB097616EE732E5
2016-05-18 00:57:25,115 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_RESULTS"]
2016-05-18 00:57:25,116 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] C544969E82FA400694A3A87C959831EE, for 1 inputs
2016-05-18 00:57:25,116 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AF72E618E89E4F238EA3F250EF1AADDD, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:25,118 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:25,118 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F8CB379FAE8A4DDF88CE5AA880A9C26D: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:25,124 INFO output.MROutput (MROutput.java:close(526)) - 968C99D1C0434F21ABB097616EE732E5 closed
2016-05-18 00:57:25,125 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533044752_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:25,126 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533044752_0001_1_01_000000_0
2016-05-18 00:57:25,127 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AF72E618E89E4F238EA3F250EF1AADDD, mem on close (mb), free: 48, total: 397, max: 672
2016-05-18 00:57:25,127 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F8CB379FAE8A4DDF88CE5AA880A9C26D
2016-05-18 00:57:25,127 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F8CB379FAE8A4DDF88CE5AA880A9C26D: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:25,127 INFO output.MROutput (MROutput.java:close(526)) - 968C99D1C0434F21ABB097616EE732E5 closed
2016-05-18 00:57:25,128 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:25 Completed running task attempt: attempt_1463533044752_0001_1_01_000000_0
2016-05-18 00:57:25,129 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533044752_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:25,129 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533044752_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:25,130 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533044752_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533045116, Process_Duration=10, Process_End_Time=1463533045126, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:25,133 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533044752_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:25,133 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:25,133 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AF72E618E89E4F238EA3F250EF1AADDD, taskAttemptId=attempt_1463533044752_0001_1_01_000000_0, creationTime=1463533045065, allocationTime=1463533045071, startTime=1463533045075, finishTime=1463533045133, timeTaken=58, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:25,133 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:25,134 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,134 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533044752_0001_1_01_000000_0
2016-05-18 00:57:25,134 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533044752_0001_00_000002
2016-05-18 00:57:25,135 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:TASK_FINISHED]: vertexName=AF72E618E89E4F238EA3F250EF1AADDD, taskId=task_1463533044752_0001_1_01_000000, startTime=1463533045075, finishTime=1463533045134, timeTaken=59, status=SUCCEEDED, successfulAttemptID=attempt_1463533044752_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533045116, Process_Duration=10, Process_End_Time=1463533045126, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:25,136 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:25,136 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD]
2016-05-18 00:57:25,137 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:VERTEX_FINISHED]: vertexName=AF72E618E89E4F238EA3F250EF1AADDD, vertexId=vertex_1463533044752_0001_1_01, initRequestedTime=1463533044937, initedTime=1463533044944, startRequestedTime=1463533044956, startedTime=1463533044956, finishTime=1463533045136, timeTaken=180, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533045116, Process_Duration=10, Process_End_Time=1463533045126, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533045075, firstTasksToStart=[ task_1463533044752_0001_1_01_000000 ], lastTaskFinishTime=1463533045133, lastTasksToFinish=[ task_1463533044752_0001_1_01_000000 ], minTaskDuration=58, maxTaskDuration=58, avgTaskDuration=58.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533044752_0001_1_01_000000 ], longestDurationTasks=[ task_1463533044752_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:25,138 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:25,138 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533044752_0001_00_000002, stoppedTime=1463533045138, exitStatus=0
2016-05-18 00:57:25,138 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:25,138 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:25,139 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533044752_0001_1_00 [F8CB379FAE8A4DDF88CE5AA880A9C26D]
2016-05-18 00:57:25,139 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533044752_0001_1
2016-05-18 00:57:25,139 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533044752_0001_1
2016-05-18 00:57:25,139 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 968C99D1C0434F21ABB097616EE732E5 for vertex: vertex_1463533044752_0001_1_01 [AF72E618E89E4F238EA3F250EF1AADDD], outputName: 968C99D1C0434F21ABB097616EE732E5
2016-05-18 00:57:25,139 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533044752_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:25,140 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533044752_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:25,146 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533044752_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:25,146 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533044752_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:25,146 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533044752_0001_00_000002, Exiting
2016-05-18 00:57:25,146 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533044752_0001_00_000002
2016-05-18 00:57:25,147 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533044752_0001_00_000002
2016-05-18 00:57:25,147 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533044752_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:25,148 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533044752_0001_00_000002 completed successfully
2016-05-18 00:57:25,149 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:968C99D1C0434F21ABB097616EE732E5 of vertex/vertexGroup:AF72E618E89E4F238EA3F250EF1AADDD isVertexGroupOutput:false
2016-05-18 00:57:25,149 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:25,151 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533044752_0001_1, queueSize=0
2016-05-18 00:57:25,152 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533044752_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533044752_0001_1, startTime=1463533044932, finishTime=1463533045149, timeTaken=217, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=440, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066090149, Process_Duration=14, Process_End_Time=2927066090163, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:25,152 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533044752_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:25,152 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533044752_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:25 Completed Dag: dag_1463533044752_0001_1
2016-05-18 00:57:25,153 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533044752_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:25,153 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533044752_0001_1_post
2016-05-18 00:57:25,153 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,153 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:25,153 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:25,153 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688] every/(1/1) ...atform/-1_'field'_RESULTS, with id=dag_1463533044752_0001_1
2016-05-18 00:57:25,153 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688] every/(1/1) ...atform/-1_'field'_RESULTS, with id=dag_1463533044752_0001_1
2016-05-18 00:57:25,164 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...atform/-1_'field'_RESULTS, applicationId=application_1463533044752_0001
2016-05-18 00:57:25,167 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:25,167 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:25,167 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:25,167 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:25,167 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:25,168 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:25,169 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:25,170 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:25,170 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:25,170 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:25,170 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:25,171 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:25,171 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:25,173 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:25,178 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/DC938B91ED764B1E91882EC0F49CDCA0/B45852B10E7745ADA2FE736062C45688/.tez/application_1463533044752_0001
2016-05-18 00:57:25,179 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_RESULTS/_temporary
2016-05-18 00:57:25,180 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.450
2016-05-18 00:57:25,285 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_'field'_RESULTS
2016-05-18 00:57:25,316 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:25,317 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:25,371 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.024
2016-05-18 00:57:25,384 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.037
2016-05-18 00:57:25,385 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:25,385 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:25,385 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:25,406 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:25,421 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_ALL
2016-05-18 00:57:25,488 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:25,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:25,490 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_ALL"]
2016-05-18 00:57:25,490 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:25,491 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:25,492 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:25,492 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/-1_'field'_ALL
2016-05-18 00:57:25,493 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:25,507 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:25,511 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:25,512 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:25,512 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:25,520 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001 doesn't exist and is created
2016-05-18 00:57:25,534 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:25,534 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:25,535 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001_wd
2016-05-18 00:57:25,547 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533045511_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:25,547 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:25,547 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:25,549 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:25,549 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:25,549 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:25,549 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:25,550 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:25,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:25,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:25,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:25,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:25,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:25,551 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:25,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:25,552 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:25,552 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:25,552 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:25,552 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:25,552 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:25,554 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:25,554 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:25,554 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:25,555 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001_wd/localmode-log-dir
2016-05-18 00:57:25,555 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001_wd/localmode-log-dir/history.txt.appattempt_1463533045511_0001_000000, maxErrors=10
2016-05-18 00:57:25,555 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:25,555 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533045511_0001_000000, appSubmitTime=1463533045546, launchTime=1463533045546
2016-05-18 00:57:25,564 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:25,569 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:60325
2016-05-18 00:57:25,569 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533045511
2016-05-18 00:57:25,571 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533045511_0001_000000, startTime=1463533045571
2016-05-18 00:57:25,571 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:25,634 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:25,634 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:25,635 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/-1_'field'_ALL, applicationId=application_1463533045511_0001, dagName=[BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD] every/(1/1) ...lyplatform/-1_'field'_ALL
2016-05-18 00:57:25,659 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD] every/(1/1) ...lyplatform/-1_'field'_ALL
2016-05-18 00:57:25,660 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:25,661 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533045511_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001_wd/localmode-log-dir/dag_1463533045511_0001_1.dot
2016-05-18 00:57:25,662 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001_wd/localmode-log-dir/dag_1463533045511_0001_1-tez-dag.pb.txt
2016-05-18 00:57:25,670 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533045511_0001_1
2016-05-18 00:57:25,670 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,670 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD] every/(1/1) ...lyplatform/-1_'field'_ALL
2016-05-18 00:57:25 Running Dag: dag_1463533045511_0001_1
2016-05-18 00:57:25,681 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533045511_0001_1, submitTime=1463533045660
2016-05-18 00:57:25,682 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001/tez-conf.pb" } size: 36913 timestamp: 1463533045000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:25,682 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,683 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:25,684 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001/tez-conf.pb" } size: 36913 timestamp: 1463533045000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:25,684 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]
2016-05-18 00:57:25,684 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:25,685 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:25,687 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533045511_0001_1, initTime=1463533045685
2016-05-18 00:57:25,687 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533045511_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:25,688 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:25,688 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:DAG_STARTED]: dagID=dag_1463533045511_0001_1, startTime=1463533045688
2016-05-18 00:57:25,688 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533045511_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:25,688 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C8FA051D4BEE4CAB9494133C0A009F4E : {73A280020084482F97CDBAD53ACAE5F7={InputName=73A280020084482F97CDBAD53ACAE5F7}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:25,689 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 73A280020084482F97CDBAD53ACAE5F7, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:25,689 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,689 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,689 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,689 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,690 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/-1_'field'_ALL, applicationId=application_1463533045511_0001, dagName=[BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD] every/(1/1) ...lyplatform/-1_'field'_ALL
2016-05-18 00:57:25,691 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:25,691 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]
2016-05-18 00:57:25,693 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 73A280020084482F97CDBAD53ACAE5F7 on vertex vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,695 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:25,695 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]
2016-05-18 00:57:25,696 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]
2016-05-18 00:57:25,698 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533045511_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:25,699 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D], numAdditionalOutputs=1
2016-05-18 00:57:25,699 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=C95D053FE40649D0B27E684578C72368, vertex=vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:25,700 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 73A280020084482F97CDBAD53ACAE5F7 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:25,703 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A71A6DE2C5E44AA882140EF2DA49CF1D:C95D053FE40649D0B27E684578C72368 using oldmapred API
2016-05-18 00:57:25,703 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:25,704 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:25,704 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:25,705 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:25,704 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=C95D053FE40649D0B27E684578C72368, vertexName=A71A6DE2C5E44AA882140EF2DA49CF1D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:25,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A71A6DE2C5E44AA882140EF2DA49CF1D, vertexId=vertex_1463533045511_0001_1_01, initRequestedTime=1463533045691, initedTime=1463533045699, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:25,707 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:25,707 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 73A280020084482F97CDBAD53ACAE5F7 on vertex vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,708 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:25,709 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {73A280020084482F97CDBAD53ACAE5F7=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:25,710 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] parallelism set to 1
2016-05-18 00:57:25,710 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:25,711 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C8FA051D4BEE4CAB9494133C0A009F4E, vertexId=vertex_1463533045511_0001_1_00, initRequestedTime=1463533045688, initedTime=1463533045711, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:25,711 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] with distanceFromRoot: 0
2016-05-18 00:57:25,711 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:25,711 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533045511_0001_1_00, startRequestedTime=1463533045691, startedTime=1463533045711
2016-05-18 00:57:25,712 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533045511_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:25,712 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:25,712 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533045511_0001_1_00 for vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D] numStartedSources: 1 numSources: 1
2016-05-18 00:57:25,712 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D] with distanceFromRoot: 1
2016-05-18 00:57:25,713 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533045511_0001_1_01, startRequestedTime=1463533045712, startedTime=1463533045712
2016-05-18 00:57:25,714 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:25,718 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A71A6DE2C5E44AA882140EF2DA49CF1D with 0 source tasks and 1 pending tasks
2016-05-18 00:57:25,719 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C8FA051D4BEE4CAB9494133C0A009F4E in vertex: A71A6DE2C5E44AA882140EF2DA49CF1D numBipartiteSourceTasks: 1
2016-05-18 00:57:25,719 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C8FA051D4BEE4CAB9494133C0A009F4E
2016-05-18 00:57:25,719 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,720 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:TASK_STARTED]: vertexName=C8FA051D4BEE4CAB9494133C0A009F4E, taskId=task_1463533045511_0001_1_00_000000, scheduledTime=1463533045720, launchTime=1463533045720
2016-05-18 00:57:25,720 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,721 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:25,723 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533045511_0001_00_000001, launchTime=1463533045723
2016-05-18 00:57:25,724 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533045511_0001_00_000001
2016-05-18 00:57:25,725 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533045511_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533045511_0001_1_00_000000_0
2016-05-18 00:57:25,725 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533045511_0001_1_00_000000_0] started. Is using containerId: [container_1463533045511_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:25,726 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C8FA051D4BEE4CAB9494133C0A009F4E, taskAttemptId=attempt_1463533045511_0001_1_00_000000_0, startTime=1463533045725, containerId=container_1463533045511_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:25 Starting to run new task attempt: attempt_1463533045511_0001_1_00_000000_0
2016-05-18 00:57:25,727 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533045511_0001_1_00_000000_0
2016-05-18 00:57:25,727 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,727 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:25,728 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD] every/(1/1) ...lyplatform/-1_'field'_ALL, VertexName: C8FA051D4BEE4CAB9494133C0A009F4E, VertexParallelism: 1, TaskAttemptID:attempt_1463533045511_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=73A280020084482F97CDBAD53ACAE5F7, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A71A6DE2C5E44AA882140EF2DA49CF1D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:25,728 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:25,729 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533045511_0001_1_00_000000_0
2016-05-18 00:57:25,730 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533045511_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:25,730 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533045511_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:25,731 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533045511_0001, with dag id: dag_1463533045511_0001_1
2016-05-18 00:57:25,735 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:25,737 INFO input.MRInput (MRInput.java:initialize(440)) - 73A280020084482F97CDBAD53ACAE5F7 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:25,738 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 73A280020084482F97CDBAD53ACAE5F7
2016-05-18 00:57:25,751 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C8FA051D4BEE4CAB9494133C0A009F4E, ordinal: 0
2016-05-18 00:57:25,751 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C8FA051D4BEE4CAB9494133C0A009F4E, mem on start (mb), free: 139, total: 377, max: 672
2016-05-18 00:57:25,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:25,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:25,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:25,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:25,752 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[73A280020084482F97CDBAD53ACAE5F7:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [A71A6DE2C5E44AA882140EF2DA49CF1D:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:25,752 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:25,752 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:25,752 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:25,752 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[73A280020084482F97CDBAD53ACAE5F7:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [A71A6DE2C5E44AA882140EF2DA49CF1D:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:25,752 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:25,752 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 73A280020084482F97CDBAD53ACAE5F7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:25,752 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:25,753 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:25,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 73A280020084482F97CDBAD53ACAE5F7
2016-05-18 00:57:25,755 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:25,756 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533045511_0001_1_00_000000_0
2016-05-18 00:57:25,759 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:25,761 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 73A280020084482F97CDBAD53ACAE5F7
2016-05-18 00:57:25,761 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: A8A9DB414B5842CF95A57F19EFDED4AE
2016-05-18 00:57:25,762 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] A8A9DB414B5842CF95A57F19EFDED4AE
2016-05-18 00:57:25,764 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:25,765 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:25,766 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 73A280020084482F97CDBAD53ACAE5F7 initialized RecordReader from event
2016-05-18 00:57:25,768 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:25,769 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A71A6DE2C5E44AA882140EF2DA49CF1D using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@85d64c8, comparator=cascading.tuple.hadoop.util.TupleComparator@2b838ac2, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:25,769 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A71A6DE2C5E44AA882140EF2DA49CF1D: , UsingHashComparator=false
2016-05-18 00:57:25,782 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:25,783 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:25,783 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A71A6DE2C5E44AA882140EF2DA49CF1D: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:25,783 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A71A6DE2C5E44AA882140EF2DA49CF1D: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:25,784 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:25,785 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C8FA051D4BEE4CAB9494133C0A009F4E, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:25,789 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533045511_0001_1_00_000000_0
2016-05-18 00:57:25,789 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C8FA051D4BEE4CAB9494133C0A009F4E, mem on close (mb), free: 37, total: 377, max: 672
2016-05-18 00:57:25,790 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A71A6DE2C5E44AA882140EF2DA49CF1D: Starting flush of map output
2016-05-18 00:57:25,790 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A71A6DE2C5E44AA882140EF2DA49CF1D: Span0.length = 20, perItem = 6
2016-05-18 00:57:25,790 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A71A6DE2C5E44AA882140EF2DA49CF1D: done sorting span=0, length=20, time=0
2016-05-18 00:57:25,791 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A71A6DE2C5E44AA882140EF2DA49CF1D: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:25,793 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A71A6DE2C5E44AA882140EF2DA49CF1D: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001_wd/localmode-local-dir/output/attempt_1463533045511_0001_1_00_000000_0_10021_0/file.out
2016-05-18 00:57:25,799 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:25 Completed running task attempt: attempt_1463533045511_0001_1_00_000000_0
2016-05-18 00:57:25,800 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533045511_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:25,800 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533045511_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:25,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533045511_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533045785, Process_Duration=4, Process_End_Time=1463533045789, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:25,802 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533045511_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:25,802 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:25,803 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:25,803 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,803 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C8FA051D4BEE4CAB9494133C0A009F4E, taskAttemptId=attempt_1463533045511_0001_1_00_000000_0, creationTime=1463533045720, allocationTime=1463533045722, startTime=1463533045725, finishTime=1463533045802, timeTaken=77, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:25,804 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533045511_0001_1_00_000000_0
2016-05-18 00:57:25,805 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533045511_0001_00_000001
2016-05-18 00:57:25,805 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:TASK_FINISHED]: vertexName=C8FA051D4BEE4CAB9494133C0A009F4E, taskId=task_1463533045511_0001_1_00_000000, startTime=1463533045725, finishTime=1463533045804, timeTaken=79, status=SUCCEEDED, successfulAttemptID=attempt_1463533045511_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533045785, Process_Duration=4, Process_End_Time=1463533045789, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:25,807 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:25,807 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,809 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:VERTEX_FINISHED]: vertexName=C8FA051D4BEE4CAB9494133C0A009F4E, vertexId=vertex_1463533045511_0001_1_00, initRequestedTime=1463533045688, initedTime=1463533045711, startRequestedTime=1463533045691, startedTime=1463533045711, finishTime=1463533045807, timeTaken=96, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533045785, Process_Duration=4, Process_End_Time=1463533045789, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533045725, firstTasksToStart=[ task_1463533045511_0001_1_00_000000 ], lastTaskFinishTime=1463533045802, lastTasksToFinish=[ task_1463533045511_0001_1_00_000000 ], minTaskDuration=77, maxTaskDuration=77, avgTaskDuration=77.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533045511_0001_1_00_000000 ], longestDurationTasks=[ task_1463533045511_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:25,809 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:25,809 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533045511_0001_00_000001, stoppedTime=1463533045809, exitStatus=0
2016-05-18 00:57:25,810 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D] attempt: attempt_1463533045511_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:25,810 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:25,810 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:25,810 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A71A6DE2C5E44AA882140EF2DA49CF1D
2016-05-18 00:57:25,812 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533045511_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:25,812 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C8FA051D4BEE4CAB9494133C0A009F4E, destinationVertexName=A71A6DE2C5E44AA882140EF2DA49CF1D. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:25,813 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]
2016-05-18 00:57:25,813 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:TASK_STARTED]: vertexName=A71A6DE2C5E44AA882140EF2DA49CF1D, taskId=task_1463533045511_0001_1_01_000000, scheduledTime=1463533045813, launchTime=1463533045813
2016-05-18 00:57:25,813 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]
2016-05-18 00:57:25,816 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533045511_0001_00_000002, launchTime=1463533045816
2016-05-18 00:57:25,817 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533045511_0001_00_000002
2016-05-18 00:57:25,817 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533045511_0001_1_01_000000_0] started. Is using containerId: [container_1463533045511_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:25,818 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A71A6DE2C5E44AA882140EF2DA49CF1D, taskAttemptId=attempt_1463533045511_0001_1_01_000000_0, startTime=1463533045817, containerId=container_1463533045511_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:25,818 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533045511_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533045511_0001_1_01_000000_0
2016-05-18 00:57:25,818 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533045511_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:25,819 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533045511_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:25,819 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533045511_0001_00_000001, Exiting
2016-05-18 00:57:25,819 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533045511_0001_00_000001
2016-05-18 00:57:25,819 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533045511_0001_00_000001
2016-05-18 00:57:25,820 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533045511_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:25,820 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533045511_0001_00_000001 completed successfully
2016-05-18 00:57:25 Starting to run new task attempt: attempt_1463533045511_0001_1_01_000000_0
2016-05-18 00:57:25,820 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533045511_0001_1_01_000000_0
2016-05-18 00:57:25,820 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,821 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:25,822 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD] every/(1/1) ...lyplatform/-1_'field'_ALL, VertexName: A71A6DE2C5E44AA882140EF2DA49CF1D, VertexParallelism: 1, TaskAttemptID:attempt_1463533045511_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C8FA051D4BEE4CAB9494133C0A009F4E, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C95D053FE40649D0B27E684578C72368, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:25,822 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:25,823 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533045511_0001_1_01_000000_0
2016-05-18 00:57:25,824 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533045511_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]
2016-05-18 00:57:25,847 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533045511_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:25,849 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:25,853 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:25,858 INFO output.MROutput (MROutput.java:initialize(421)) - C95D053FE40649D0B27E684578C72368: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:25,860 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A71A6DE2C5E44AA882140EF2DA49CF1D, ordinal: 1
2016-05-18 00:57:25,860 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A71A6DE2C5E44AA882140EF2DA49CF1D, mem on start (mb), free: 246, total: 387, max: 672
2016-05-18 00:57:25,860 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:25,861 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:25,861 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:25,861 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:25,861 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[C8FA051D4BEE4CAB9494133C0A009F4E:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [C95D053FE40649D0B27E684578C72368:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:25,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:25,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:25,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:25,862 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[C8FA051D4BEE4CAB9494133C0A009F4E:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [C95D053FE40649D0B27E684578C72368:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:25,862 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:25,863 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C8FA051D4BEE4CAB9494133C0A009F4E: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:25,863 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C8FA051D4BEE4CAB9494133C0A009F4E: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:25,864 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C8FA051D4BEE4CAB9494133C0A009F4E with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:25,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C8FA051D4BEE4CAB9494133C0A009F4E
2016-05-18 00:57:25,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C8FA051D4BEE4CAB9494133C0A009F4E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:25,866 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:25,866 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:25,866 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:25,867 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533045511_0001_1_01_000000_0
2016-05-18 00:57:25,867 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C8FA051D4BEE4CAB9494133C0A009F4E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:25,870 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533045511_0001_1_00_000000_0_10021} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533045870, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:25,870 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C8FA051D4BEE4CAB9494133C0A009F4E
2016-05-18 00:57:25,871 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:25,871 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:25,871 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C8FA051D4BEE4CAB9494133C0A009F4E, wasInterrupted=false
2016-05-18 00:57:25,871 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: A8A9DB414B5842CF95A57F19EFDED4AE
2016-05-18 00:57:25,874 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:25,874 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_ALL"], id: C95D053FE40649D0B27E684578C72368
2016-05-18 00:57:25,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:25,874 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_ALL"]
2016-05-18 00:57:25,875 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] A8A9DB414B5842CF95A57F19EFDED4AE, for 1 inputs
2016-05-18 00:57:25,877 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:25,877 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001_wd/localmode-local-dir/output/attempt_1463533045511_0001_1_00_000000_0_10021/file.out, len=181, isLocal=true
2016-05-18 00:57:25,878 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:25,878 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:25,878 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:25,879 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:25,879 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C8FA051D4BEE4CAB9494133C0A009F4E
2016-05-18 00:57:25,879 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A71A6DE2C5E44AA882140EF2DA49CF1D, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:25,879 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C8FA051D4BEE4CAB9494133C0A009F4E: Shuffle Runner thread complete
2016-05-18 00:57:25,881 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:25,881 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C8FA051D4BEE4CAB9494133C0A009F4E: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:25,887 INFO output.MROutput (MROutput.java:close(526)) - C95D053FE40649D0B27E684578C72368 closed
2016-05-18 00:57:25,889 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533045511_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:25,890 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533045511_0001_1_01_000000_0
2016-05-18 00:57:25,891 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A71A6DE2C5E44AA882140EF2DA49CF1D, mem on close (mb), free: 242, total: 387, max: 672
2016-05-18 00:57:25,891 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C8FA051D4BEE4CAB9494133C0A009F4E
2016-05-18 00:57:25,891 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C8FA051D4BEE4CAB9494133C0A009F4E: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:25,892 INFO output.MROutput (MROutput.java:close(526)) - C95D053FE40649D0B27E684578C72368 closed
2016-05-18 00:57:25,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:25 Completed running task attempt: attempt_1463533045511_0001_1_01_000000_0
2016-05-18 00:57:25,893 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533045511_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:25,894 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533045511_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:25,894 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533045511_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533045879, Process_Duration=11, Process_End_Time=1463533045890, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:25,896 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533045511_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:25,896 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:25,896 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:25,896 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A71A6DE2C5E44AA882140EF2DA49CF1D, taskAttemptId=attempt_1463533045511_0001_1_01_000000_0, creationTime=1463533045813, allocationTime=1463533045815, startTime=1463533045817, finishTime=1463533045896, timeTaken=79, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:25,897 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,897 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533045511_0001_1_01_000000_0
2016-05-18 00:57:25,897 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533045511_0001_00_000002
2016-05-18 00:57:25,898 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:TASK_FINISHED]: vertexName=A71A6DE2C5E44AA882140EF2DA49CF1D, taskId=task_1463533045511_0001_1_01_000000, startTime=1463533045817, finishTime=1463533045897, timeTaken=80, status=SUCCEEDED, successfulAttemptID=attempt_1463533045511_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533045879, Process_Duration=11, Process_End_Time=1463533045890, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:25,898 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:25,898 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D]
2016-05-18 00:57:25,900 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:VERTEX_FINISHED]: vertexName=A71A6DE2C5E44AA882140EF2DA49CF1D, vertexId=vertex_1463533045511_0001_1_01, initRequestedTime=1463533045691, initedTime=1463533045699, startRequestedTime=1463533045712, startedTime=1463533045712, finishTime=1463533045899, timeTaken=187, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533045879, Process_Duration=11, Process_End_Time=1463533045890, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533045817, firstTasksToStart=[ task_1463533045511_0001_1_01_000000 ], lastTaskFinishTime=1463533045896, lastTasksToFinish=[ task_1463533045511_0001_1_01_000000 ], minTaskDuration=79, maxTaskDuration=79, avgTaskDuration=79.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533045511_0001_1_01_000000 ], longestDurationTasks=[ task_1463533045511_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:25,900 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:25,900 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533045511_0001_00_000002, stoppedTime=1463533045900, exitStatus=0
2016-05-18 00:57:25,901 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:25,901 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:25,901 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533045511_0001_1_00 [C8FA051D4BEE4CAB9494133C0A009F4E]
2016-05-18 00:57:25,901 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533045511_0001_1
2016-05-18 00:57:25,901 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533045511_0001_1
2016-05-18 00:57:25,901 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: C95D053FE40649D0B27E684578C72368 for vertex: vertex_1463533045511_0001_1_01 [A71A6DE2C5E44AA882140EF2DA49CF1D], outputName: C95D053FE40649D0B27E684578C72368
2016-05-18 00:57:25,902 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533045511_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:25,904 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533045511_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:25,908 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533045511_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:25,908 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533045511_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:25,909 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533045511_0001_00_000002, Exiting
2016-05-18 00:57:25,909 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533045511_0001_00_000002
2016-05-18 00:57:25,909 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533045511_0001_00_000002
2016-05-18 00:57:25,909 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533045511_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:25,909 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533045511_0001_00_000002 completed successfully
2016-05-18 00:57:25,909 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:C95D053FE40649D0B27E684578C72368 of vertex/vertexGroup:A71A6DE2C5E44AA882140EF2DA49CF1D isVertexGroupOutput:false
2016-05-18 00:57:25,910 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:25,917 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533045511_0001_1, queueSize=0
2016-05-18 00:57:25,919 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533045511_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533045511_0001_1, startTime=1463533045688, finishTime=1463533045910, timeTaken=222, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=490, AM_GC_TIME_MILLIS=20, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066091664, Process_Duration=15, Process_End_Time=2927066091679, Read_Duration=4, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:25,919 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533045511_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:25,919 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533045511_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:25 Completed Dag: dag_1463533045511_0001_1
2016-05-18 00:57:25,920 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533045511_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:25,920 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533045511_0001_1_post
2016-05-18 00:57:25,920 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:25,921 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:25,921 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:25,921 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD] every/(1/1) ...lyplatform/-1_'field'_ALL, with id=dag_1463533045511_0001_1
2016-05-18 00:57:25,921 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD] every/(1/1) ...lyplatform/-1_'field'_ALL, with id=dag_1463533045511_0001_1
2016-05-18 00:57:25,927 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/-1_'field'_ALL, applicationId=application_1463533045511_0001
2016-05-18 00:57:25,931 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:25,932 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:25,932 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:25,932 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:25,932 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:25,933 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:25,933 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:25,934 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:25,934 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:25,935 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:25,935 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:25,936 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:25,936 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:25,937 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:25,941 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BE62ED0896B94485B99EF901597D5DF1/C25B975D45144C5AA57E47BB5425EFBD/.tez/application_1463533045511_0001
2016-05-18 00:57:25,942 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_ALL/_temporary
2016-05-18 00:57:25,942 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.453
2016-05-18 00:57:26,028 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_'field'_ALL
2016-05-18 00:57:26,058 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:26,058 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:26,105 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.008
2016-05-18 00:57:26,111 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.014
2016-05-18 00:57:26,111 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:26,112 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:26,113 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_'field'_REPLACE
2016-05-18 00:57:26,139 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:26,140 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:26,193 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.022
2016-05-18 00:57:26,203 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.033
2016-05-18 00:57:26,204 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:26,204 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:26,204 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:26,222 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:26,230 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_SWAP
2016-05-18 00:57:26,287 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:26,289 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:26,289 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_SWAP"]
2016-05-18 00:57:26,289 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:26,289 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:26,289 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:26,290 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...yplatform/-1_'field'_SWAP
2016-05-18 00:57:26,290 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:26,305 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:26,309 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:26,310 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:26,310 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:26,319 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001 doesn't exist and is created
2016-05-18 00:57:26,334 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:26,334 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:26,335 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001_wd
2016-05-18 00:57:26,348 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533046310_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:26,349 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:26,349 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:26,351 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:26,351 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:26,351 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:26,351 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:26,352 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:26,354 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:26,354 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:26,354 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:26,354 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:26,354 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:26,355 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:26,355 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:26,355 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:26,355 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:26,355 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:26,356 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:26,356 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:26,357 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:26,358 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:26,358 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:26,358 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001_wd/localmode-log-dir
2016-05-18 00:57:26,359 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001_wd/localmode-log-dir/history.txt.appattempt_1463533046310_0001_000000, maxErrors=10
2016-05-18 00:57:26,359 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:26,359 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533046310_0001_000000, appSubmitTime=1463533046347, launchTime=1463533046348
2016-05-18 00:57:26,365 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:26,372 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:36440
2016-05-18 00:57:26,373 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533046310
2016-05-18 00:57:26,374 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533046310_0001_000000, startTime=1463533046374
2016-05-18 00:57:26,374 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:26,434 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:26,435 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:26,435 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...yplatform/-1_'field'_SWAP, applicationId=application_1463533046310_0001, dagName=[F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E] every/(1/1) ...yplatform/-1_'field'_SWAP
2016-05-18 00:57:26,453 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E] every/(1/1) ...yplatform/-1_'field'_SWAP
2016-05-18 00:57:26,453 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:26,454 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533046310_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001_wd/localmode-log-dir/dag_1463533046310_0001_1.dot
2016-05-18 00:57:26,456 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001_wd/localmode-log-dir/dag_1463533046310_0001_1-tez-dag.pb.txt
2016-05-18 00:57:26,462 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533046310_0001_1
2016-05-18 00:57:26,463 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:26,463 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E] every/(1/1) ...yplatform/-1_'field'_SWAP
2016-05-18 00:57:26 Running Dag: dag_1463533046310_0001_1
2016-05-18 00:57:26,473 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533046310_0001_1, submitTime=1463533046453
2016-05-18 00:57:26,474 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001/tez-conf.pb" } size: 36913 timestamp: 1463533046000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:26,474 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,475 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:26,476 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001/tez-conf.pb" } size: 36913 timestamp: 1463533046000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:26,476 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]
2016-05-18 00:57:26,476 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:26,478 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:26,478 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533046310_0001_1, initTime=1463533046478
2016-05-18 00:57:26,478 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533046310_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:26,481 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:26,481 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:DAG_STARTED]: dagID=dag_1463533046310_0001_1, startTime=1463533046481
2016-05-18 00:57:26,481 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533046310_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:26,481 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D0B4CA0A7E4A42578582E557B9675B3F : {8F9C3262C04647378E7210E1EAB4FD69={InputName=8F9C3262C04647378E7210E1EAB4FD69}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:26,481 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 8F9C3262C04647378E7210E1EAB4FD69, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:26,482 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,482 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...yplatform/-1_'field'_SWAP, applicationId=application_1463533046310_0001, dagName=[F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E] every/(1/1) ...yplatform/-1_'field'_SWAP
2016-05-18 00:57:26,482 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,482 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,482 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,483 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:26,483 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]
2016-05-18 00:57:26,483 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 8F9C3262C04647378E7210E1EAB4FD69 on vertex vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,486 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 8F9C3262C04647378E7210E1EAB4FD69 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:26,487 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:26,487 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]
2016-05-18 00:57:26,488 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]
2016-05-18 00:57:26,488 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533046310_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:26,488 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B], numAdditionalOutputs=1
2016-05-18 00:57:26,488 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=120CAB05934F4194A74C8BA132D33E8E, vertex=vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:26,490 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:26,490 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:26,491 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:26,491 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:26,492 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 8F9C3262C04647378E7210E1EAB4FD69 on vertex vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,493 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DED156CDF0EF41C6B78FC11336CF2E6B:120CAB05934F4194A74C8BA132D33E8E using oldmapred API
2016-05-18 00:57:26,493 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=120CAB05934F4194A74C8BA132D33E8E, vertexName=DED156CDF0EF41C6B78FC11336CF2E6B, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:26,494 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DED156CDF0EF41C6B78FC11336CF2E6B, vertexId=vertex_1463533046310_0001_1_01, initRequestedTime=1463533046483, initedTime=1463533046488, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:26,494 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:26,495 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:26,495 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {8F9C3262C04647378E7210E1EAB4FD69=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:26,496 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] parallelism set to 1
2016-05-18 00:57:26,496 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:26,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D0B4CA0A7E4A42578582E557B9675B3F, vertexId=vertex_1463533046310_0001_1_00, initRequestedTime=1463533046481, initedTime=1463533046496, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:26,497 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] with distanceFromRoot: 0
2016-05-18 00:57:26,497 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:26,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533046310_0001_1_00, startRequestedTime=1463533046483, startedTime=1463533046497
2016-05-18 00:57:26,498 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533046310_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:26,498 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:26,498 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533046310_0001_1_00 for vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B] numStartedSources: 1 numSources: 1
2016-05-18 00:57:26,498 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B] with distanceFromRoot: 1
2016-05-18 00:57:26,499 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D0B4CA0A7E4A42578582E557B9675B3F
2016-05-18 00:57:26,499 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,499 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533046310_0001_1_01, startRequestedTime=1463533046498, startedTime=1463533046499
2016-05-18 00:57:26,499 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:26,501 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:TASK_STARTED]: vertexName=D0B4CA0A7E4A42578582E557B9675B3F, taskId=task_1463533046310_0001_1_00_000000, scheduledTime=1463533046501, launchTime=1463533046501
2016-05-18 00:57:26,502 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,503 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DED156CDF0EF41C6B78FC11336CF2E6B with 0 source tasks and 1 pending tasks
2016-05-18 00:57:26,503 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:26,505 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D0B4CA0A7E4A42578582E557B9675B3F in vertex: DED156CDF0EF41C6B78FC11336CF2E6B numBipartiteSourceTasks: 1
2016-05-18 00:57:26,507 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533046310_0001_00_000001, launchTime=1463533046507
2016-05-18 00:57:26,507 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533046310_0001_00_000001
2016-05-18 00:57:26,508 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533046310_0001_1_00_000000_0] started. Is using containerId: [container_1463533046310_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:26,509 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D0B4CA0A7E4A42578582E557B9675B3F, taskAttemptId=attempt_1463533046310_0001_1_00_000000_0, startTime=1463533046508, containerId=container_1463533046310_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:26,509 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533046310_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533046310_0001_1_00_000000_0
2016-05-18 00:57:26 Starting to run new task attempt: attempt_1463533046310_0001_1_00_000000_0
2016-05-18 00:57:26,510 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533046310_0001_1_00_000000_0
2016-05-18 00:57:26,510 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:26,510 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533046310_0001, with dag id: dag_1463533046310_0001_1
2016-05-18 00:57:26,510 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:26,515 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E] every/(1/1) ...yplatform/-1_'field'_SWAP, VertexName: D0B4CA0A7E4A42578582E557B9675B3F, VertexParallelism: 1, TaskAttemptID:attempt_1463533046310_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=8F9C3262C04647378E7210E1EAB4FD69, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DED156CDF0EF41C6B78FC11336CF2E6B, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:26,515 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:26,518 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533046310_0001_1_00_000000_0
2016-05-18 00:57:26,518 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533046310_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:26,520 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533046310_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:26,522 INFO input.MRInput (MRInput.java:initialize(440)) - 8F9C3262C04647378E7210E1EAB4FD69 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:26,522 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 8F9C3262C04647378E7210E1EAB4FD69
2016-05-18 00:57:26,522 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:26,529 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D0B4CA0A7E4A42578582E557B9675B3F, ordinal: 0
2016-05-18 00:57:26,529 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D0B4CA0A7E4A42578582E557B9675B3F, mem on start (mb), free: 112, total: 387, max: 672
2016-05-18 00:57:26,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:26,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:26,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:26,530 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:26,530 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[8F9C3262C04647378E7210E1EAB4FD69:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [DED156CDF0EF41C6B78FC11336CF2E6B:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:26,531 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:26,531 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:26,531 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:26,532 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[8F9C3262C04647378E7210E1EAB4FD69:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [DED156CDF0EF41C6B78FC11336CF2E6B:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:26,532 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:26,532 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 8F9C3262C04647378E7210E1EAB4FD69 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:26,533 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:26,533 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:26,532 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 8F9C3262C04647378E7210E1EAB4FD69
2016-05-18 00:57:26,533 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:26,534 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533046310_0001_1_00_000000_0
2016-05-18 00:57:26,538 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:26,539 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 8F9C3262C04647378E7210E1EAB4FD69
2016-05-18 00:57:26,539 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: AE455AF99BAD451EAE50F14046043C5B
2016-05-18 00:57:26,539 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] AE455AF99BAD451EAE50F14046043C5B
2016-05-18 00:57:26,540 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:26,540 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 8F9C3262C04647378E7210E1EAB4FD69 initialized RecordReader from event
2016-05-18 00:57:26,541 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:26,544 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:26,545 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DED156CDF0EF41C6B78FC11336CF2E6B using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6c6e86b7, comparator=cascading.tuple.hadoop.util.TupleComparator@4c4e8ffe, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:26,545 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DED156CDF0EF41C6B78FC11336CF2E6B: , UsingHashComparator=false
2016-05-18 00:57:26,573 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:26,573 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:26,573 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DED156CDF0EF41C6B78FC11336CF2E6B: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:26,574 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DED156CDF0EF41C6B78FC11336CF2E6B: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:26,574 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:26,574 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D0B4CA0A7E4A42578582E557B9675B3F, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:26,577 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533046310_0001_1_00_000000_0
2016-05-18 00:57:26,577 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D0B4CA0A7E4A42578582E557B9675B3F, mem on close (mb), free: 142, total: 385, max: 672
2016-05-18 00:57:26,579 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DED156CDF0EF41C6B78FC11336CF2E6B: Starting flush of map output
2016-05-18 00:57:26,580 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DED156CDF0EF41C6B78FC11336CF2E6B: Span0.length = 20, perItem = 6
2016-05-18 00:57:26,580 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DED156CDF0EF41C6B78FC11336CF2E6B: done sorting span=0, length=20, time=0
2016-05-18 00:57:26,581 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DED156CDF0EF41C6B78FC11336CF2E6B: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:26,582 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DED156CDF0EF41C6B78FC11336CF2E6B: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001_wd/localmode-local-dir/output/attempt_1463533046310_0001_1_00_000000_0_10027_0/file.out
2016-05-18 00:57:26,587 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:26 Completed running task attempt: attempt_1463533046310_0001_1_00_000000_0
2016-05-18 00:57:26,588 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533046310_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:26,588 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533046310_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:26,589 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533046310_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533046575, Process_Duration=2, Process_End_Time=1463533046577, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:26,590 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533046310_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:26,590 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:26,590 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:26,590 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:26,590 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533046310_0001_00_000001
2016-05-18 00:57:26,590 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D0B4CA0A7E4A42578582E557B9675B3F, taskAttemptId=attempt_1463533046310_0001_1_00_000000_0, creationTime=1463533046502, allocationTime=1463533046506, startTime=1463533046508, finishTime=1463533046590, timeTaken=82, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:26,591 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533046310_0001_1_00_000000_0
2016-05-18 00:57:26,593 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:TASK_FINISHED]: vertexName=D0B4CA0A7E4A42578582E557B9675B3F, taskId=task_1463533046310_0001_1_00_000000, startTime=1463533046508, finishTime=1463533046591, timeTaken=83, status=SUCCEEDED, successfulAttemptID=attempt_1463533046310_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533046575, Process_Duration=2, Process_End_Time=1463533046577, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:26,593 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:26,593 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,595 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:VERTEX_FINISHED]: vertexName=D0B4CA0A7E4A42578582E557B9675B3F, vertexId=vertex_1463533046310_0001_1_00, initRequestedTime=1463533046481, initedTime=1463533046496, startRequestedTime=1463533046483, startedTime=1463533046497, finishTime=1463533046594, timeTaken=97, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533046575, Process_Duration=2, Process_End_Time=1463533046577, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533046508, firstTasksToStart=[ task_1463533046310_0001_1_00_000000 ], lastTaskFinishTime=1463533046590, lastTasksToFinish=[ task_1463533046310_0001_1_00_000000 ], minTaskDuration=82, maxTaskDuration=82, avgTaskDuration=82.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533046310_0001_1_00_000000 ], longestDurationTasks=[ task_1463533046310_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:26,595 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:26,596 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533046310_0001_00_000001, stoppedTime=1463533046596, exitStatus=0
2016-05-18 00:57:26,596 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B] attempt: attempt_1463533046310_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:26,596 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:26,596 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DED156CDF0EF41C6B78FC11336CF2E6B
2016-05-18 00:57:26,596 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:26,597 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D0B4CA0A7E4A42578582E557B9675B3F, destinationVertexName=DED156CDF0EF41C6B78FC11336CF2E6B. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:26,597 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533046310_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:26,597 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]
2016-05-18 00:57:26,597 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:TASK_STARTED]: vertexName=DED156CDF0EF41C6B78FC11336CF2E6B, taskId=task_1463533046310_0001_1_01_000000, scheduledTime=1463533046597, launchTime=1463533046597
2016-05-18 00:57:26,598 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]
2016-05-18 00:57:26,599 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533046310_0001_00_000002, launchTime=1463533046599
2016-05-18 00:57:26,600 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533046310_0001_00_000002
2016-05-18 00:57:26,600 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533046310_0001_1_01_000000_0] started. Is using containerId: [container_1463533046310_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:26,600 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DED156CDF0EF41C6B78FC11336CF2E6B, taskAttemptId=attempt_1463533046310_0001_1_01_000000_0, startTime=1463533046600, containerId=container_1463533046310_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:26,601 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533046310_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533046310_0001_1_01_000000_0
2016-05-18 00:57:26,601 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533046310_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:26 Starting to run new task attempt: attempt_1463533046310_0001_1_01_000000_0
2016-05-18 00:57:26,602 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533046310_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:26,602 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533046310_0001_1_01_000000_0
2016-05-18 00:57:26,602 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:26,602 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533046310_0001_00_000001, Exiting
2016-05-18 00:57:26,602 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:26,602 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533046310_0001_00_000001
2016-05-18 00:57:26,603 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E] every/(1/1) ...yplatform/-1_'field'_SWAP, VertexName: DED156CDF0EF41C6B78FC11336CF2E6B, VertexParallelism: 1, TaskAttemptID:attempt_1463533046310_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D0B4CA0A7E4A42578582E557B9675B3F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=120CAB05934F4194A74C8BA132D33E8E, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:26,603 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533046310_0001_00_000001
2016-05-18 00:57:26,603 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:26,604 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533046310_0001_1_01_000000_0
2016-05-18 00:57:26,604 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533046310_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:26,605 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533046310_0001_00_000001 completed successfully
2016-05-18 00:57:26,606 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533046310_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]
2016-05-18 00:57:26,606 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533046310_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:26,612 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:26,613 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:26,615 INFO output.MROutput (MROutput.java:initialize(421)) - 120CAB05934F4194A74C8BA132D33E8E: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:26,620 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DED156CDF0EF41C6B78FC11336CF2E6B, ordinal: 1
2016-05-18 00:57:26,621 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DED156CDF0EF41C6B78FC11336CF2E6B, mem on start (mb), free: 136, total: 385, max: 672
2016-05-18 00:57:26,621 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:26,621 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:26,621 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:26,621 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:26,622 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[120CAB05934F4194A74C8BA132D33E8E:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [D0B4CA0A7E4A42578582E557B9675B3F:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:26,622 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:26,622 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:26,623 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:26,623 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[120CAB05934F4194A74C8BA132D33E8E:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [D0B4CA0A7E4A42578582E557B9675B3F:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:26,623 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:26,623 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D0B4CA0A7E4A42578582E557B9675B3F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:26,624 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:26,624 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:26,624 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D0B4CA0A7E4A42578582E557B9675B3F: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:26,624 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D0B4CA0A7E4A42578582E557B9675B3F: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:26,625 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D0B4CA0A7E4A42578582E557B9675B3F with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:26,626 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D0B4CA0A7E4A42578582E557B9675B3F
2016-05-18 00:57:26,626 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:26,627 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533046310_0001_1_01_000000_0
2016-05-18 00:57:26,630 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:26,630 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D0B4CA0A7E4A42578582E557B9675B3F: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:26,631 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: AE455AF99BAD451EAE50F14046043C5B
2016-05-18 00:57:26,632 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533046310_0001_1_00_000000_0_10027} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533046632, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:26,632 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D0B4CA0A7E4A42578582E557B9675B3F
2016-05-18 00:57:26,633 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:26,632 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_SWAP"], id: 120CAB05934F4194A74C8BA132D33E8E
2016-05-18 00:57:26,633 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D0B4CA0A7E4A42578582E557B9675B3F, wasInterrupted=false
2016-05-18 00:57:26,635 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_SWAP"]
2016-05-18 00:57:26,635 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] AE455AF99BAD451EAE50F14046043C5B, for 1 inputs
2016-05-18 00:57:26,635 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:26,636 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:26,638 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:26,638 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001_wd/localmode-local-dir/output/attempt_1463533046310_0001_1_00_000000_0_10027/file.out, len=181, isLocal=true
2016-05-18 00:57:26,638 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:26,638 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:26,638 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:26,639 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:26,639 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D0B4CA0A7E4A42578582E557B9675B3F
2016-05-18 00:57:26,639 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D0B4CA0A7E4A42578582E557B9675B3F: Shuffle Runner thread complete
2016-05-18 00:57:26,639 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DED156CDF0EF41C6B78FC11336CF2E6B, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:26,641 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:26,641 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D0B4CA0A7E4A42578582E557B9675B3F: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:26,648 INFO output.MROutput (MROutput.java:close(526)) - 120CAB05934F4194A74C8BA132D33E8E closed
2016-05-18 00:57:26,649 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533046310_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:26,650 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533046310_0001_1_01_000000_0
2016-05-18 00:57:26,650 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DED156CDF0EF41C6B78FC11336CF2E6B, mem on close (mb), free: 133, total: 385, max: 672
2016-05-18 00:57:26,650 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D0B4CA0A7E4A42578582E557B9675B3F
2016-05-18 00:57:26,651 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D0B4CA0A7E4A42578582E557B9675B3F: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:26,651 INFO output.MROutput (MROutput.java:close(526)) - 120CAB05934F4194A74C8BA132D33E8E closed
2016-05-18 00:57:26,651 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:26 Completed running task attempt: attempt_1463533046310_0001_1_01_000000_0
2016-05-18 00:57:26,652 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533046310_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:26,653 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533046310_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:26,653 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533046310_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533046640, Process_Duration=10, Process_End_Time=1463533046650, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:26,655 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533046310_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:26,655 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DED156CDF0EF41C6B78FC11336CF2E6B, taskAttemptId=attempt_1463533046310_0001_1_01_000000_0, creationTime=1463533046598, allocationTime=1463533046598, startTime=1463533046600, finishTime=1463533046655, timeTaken=55, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:26,655 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:26,655 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:26,655 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533046310_0001_1_01_000000_0
2016-05-18 00:57:26,656 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:26,656 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533046310_0001_00_000002
2016-05-18 00:57:26,656 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:TASK_FINISHED]: vertexName=DED156CDF0EF41C6B78FC11336CF2E6B, taskId=task_1463533046310_0001_1_01_000000, startTime=1463533046600, finishTime=1463533046655, timeTaken=55, status=SUCCEEDED, successfulAttemptID=attempt_1463533046310_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533046640, Process_Duration=10, Process_End_Time=1463533046650, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:26,657 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:26,657 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B]
2016-05-18 00:57:26,658 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:VERTEX_FINISHED]: vertexName=DED156CDF0EF41C6B78FC11336CF2E6B, vertexId=vertex_1463533046310_0001_1_01, initRequestedTime=1463533046483, initedTime=1463533046488, startRequestedTime=1463533046498, startedTime=1463533046499, finishTime=1463533046657, timeTaken=158, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533046640, Process_Duration=10, Process_End_Time=1463533046650, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533046600, firstTasksToStart=[ task_1463533046310_0001_1_01_000000 ], lastTaskFinishTime=1463533046655, lastTasksToFinish=[ task_1463533046310_0001_1_01_000000 ], minTaskDuration=55, maxTaskDuration=55, avgTaskDuration=55.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533046310_0001_1_01_000000 ], longestDurationTasks=[ task_1463533046310_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:26,658 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:26,659 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533046310_0001_00_000002, stoppedTime=1463533046659, exitStatus=0
2016-05-18 00:57:26,659 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:26,659 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:26,659 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533046310_0001_1_00 [D0B4CA0A7E4A42578582E557B9675B3F]
2016-05-18 00:57:26,659 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533046310_0001_1
2016-05-18 00:57:26,660 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533046310_0001_1
2016-05-18 00:57:26,661 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 120CAB05934F4194A74C8BA132D33E8E for vertex: vertex_1463533046310_0001_1_01 [DED156CDF0EF41C6B78FC11336CF2E6B], outputName: 120CAB05934F4194A74C8BA132D33E8E
2016-05-18 00:57:26,664 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533046310_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:26,664 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533046310_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:26,665 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533046310_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:26,665 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533046310_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:26,665 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533046310_0001_00_000002, Exiting
2016-05-18 00:57:26,665 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533046310_0001_00_000002
2016-05-18 00:57:26,666 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533046310_0001_00_000002
2016-05-18 00:57:26,666 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533046310_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:26,666 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533046310_0001_00_000002 completed successfully
2016-05-18 00:57:26,668 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:120CAB05934F4194A74C8BA132D33E8E of vertex/vertexGroup:DED156CDF0EF41C6B78FC11336CF2E6B isVertexGroupOutput:false
2016-05-18 00:57:26,668 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:26,672 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533046310_0001_1, queueSize=0
2016-05-18 00:57:26,672 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046310_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533046310_0001_1, startTime=1463533046481, finishTime=1463533046668, timeTaken=187, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=380, AM_GC_TIME_MILLIS=15, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066093215, Process_Duration=12, Process_End_Time=2927066093227, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:26,672 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533046310_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:26,673 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533046310_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:26 Completed Dag: dag_1463533046310_0001_1
2016-05-18 00:57:26,673 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533046310_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:26,673 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533046310_0001_1_post
2016-05-18 00:57:26,673 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:26,673 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:26,674 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:26,674 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E] every/(1/1) ...yplatform/-1_'field'_SWAP, with id=dag_1463533046310_0001_1
2016-05-18 00:57:26,674 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E] every/(1/1) ...yplatform/-1_'field'_SWAP, with id=dag_1463533046310_0001_1
2016-05-18 00:57:26,682 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...yplatform/-1_'field'_SWAP, applicationId=application_1463533046310_0001
2016-05-18 00:57:26,684 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:26,684 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:26,684 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:26,684 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:26,684 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:26,684 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:26,685 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:26,685 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:26,685 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:26,685 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:26,685 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:26,686 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:26,686 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:26,687 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:26,690 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F9E61262D07647FE8FBE3116069BBC6A/A66F09F5F2324563A946CB6106DDC88E/.tez/application_1463533046310_0001
2016-05-18 00:57:26,690 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_'field'_SWAP/_temporary
2016-05-18 00:57:26,691 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.403
2016-05-18 00:57:26,767 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_'field'_SWAP
2016-05-18 00:57:26,794 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:26,794 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:26,841 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.015
2016-05-18 00:57:26,862 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.036
2016-05-18 00:57:26,863 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:26,863 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:26,864 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:26,886 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:26,894 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_-1
2016-05-18 00:57:26,949 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:26,949 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:26,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_-1"]
2016-05-18 00:57:26,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:26,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:26,951 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:26,952 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...blyplatform/-1_UNKNOWN_-1
2016-05-18 00:57:26,952 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:26,967 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:26,973 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:26,973 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:26,974 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:26,984 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001 doesn't exist and is created
2016-05-18 00:57:27,000 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:27,000 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:27,002 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001_wd
2016-05-18 00:57:27,012 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533046973_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:27,012 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:27,012 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:27,014 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:27,014 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:27,014 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:27,014 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:27,016 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:27,018 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:27,018 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:27,019 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:27,019 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:27,022 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:27,023 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:27,023 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:27,024 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001_wd/localmode-log-dir
2016-05-18 00:57:27,024 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001_wd/localmode-log-dir/history.txt.appattempt_1463533046973_0001_000000, maxErrors=10
2016-05-18 00:57:27,024 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:27,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533046973_0001_000000, appSubmitTime=1463533047011, launchTime=1463533047011
2016-05-18 00:57:27,029 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:27,034 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:59085
2016-05-18 00:57:27,035 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533046973
2016-05-18 00:57:27,037 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533046973_0001_000000, startTime=1463533047037
2016-05-18 00:57:27,037 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:27,101 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:27,101 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:27,101 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...blyplatform/-1_UNKNOWN_-1, applicationId=application_1463533046973_0001, dagName=[4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030] every/(1/1) ...blyplatform/-1_UNKNOWN_-1
2016-05-18 00:57:27,120 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030] every/(1/1) ...blyplatform/-1_UNKNOWN_-1
2016-05-18 00:57:27,120 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:27,121 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533046973_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001_wd/localmode-log-dir/dag_1463533046973_0001_1.dot
2016-05-18 00:57:27,122 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001_wd/localmode-log-dir/dag_1463533046973_0001_1-tez-dag.pb.txt
2016-05-18 00:57:27,134 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533046973_0001_1
2016-05-18 00:57:27,134 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:27,134 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030] every/(1/1) ...blyplatform/-1_UNKNOWN_-1
2016-05-18 00:57:27 Running Dag: dag_1463533046973_0001_1
2016-05-18 00:57:27,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533046973_0001_1, submitTime=1463533047120
2016-05-18 00:57:27,147 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001/tez-conf.pb" } size: 36913 timestamp: 1463533046000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:27,147 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,147 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:27,148 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001/tez-conf.pb" } size: 36913 timestamp: 1463533046000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:27,148 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]
2016-05-18 00:57:27,148 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:27,148 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:27,149 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533046973_0001_1, initTime=1463533047149
2016-05-18 00:57:27,150 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533046973_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:27,150 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:27,150 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:DAG_STARTED]: dagID=dag_1463533046973_0001_1, startTime=1463533047150
2016-05-18 00:57:27,150 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533046973_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:27,151 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E004C2E204794287A7B15EF0E9697CEC : {BE61D95195A2460A98450799B3E79BA0={InputName=BE61D95195A2460A98450799B3E79BA0}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:27,151 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: BE61D95195A2460A98450799B3E79BA0, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:27,151 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,152 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,152 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,152 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,152 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...blyplatform/-1_UNKNOWN_-1, applicationId=application_1463533046973_0001, dagName=[4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030] every/(1/1) ...blyplatform/-1_UNKNOWN_-1
2016-05-18 00:57:27,155 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:27,155 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]
2016-05-18 00:57:27,159 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:27,159 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]
2016-05-18 00:57:27,159 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]
2016-05-18 00:57:27,159 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533046973_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:27,159 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8], numAdditionalOutputs=1
2016-05-18 00:57:27,160 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=6AD48C4884894F16AC2FB1B6A591DA0A, vertex=vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:27,161 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: BE61D95195A2460A98450799B3E79BA0 on vertex vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,163 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input BE61D95195A2460A98450799B3E79BA0 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:27,164 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for ABA91BBB17DF456AB4B966F6327F8FB8:6AD48C4884894F16AC2FB1B6A591DA0A using oldmapred API
2016-05-18 00:57:27,165 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=6AD48C4884894F16AC2FB1B6A591DA0A, vertexName=ABA91BBB17DF456AB4B966F6327F8FB8, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:27,166 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ABA91BBB17DF456AB4B966F6327F8FB8, vertexId=vertex_1463533046973_0001_1_01, initRequestedTime=1463533047155, initedTime=1463533047159, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:27,166 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:27,167 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:27,167 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:27,167 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:27,168 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:27,168 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: BE61D95195A2460A98450799B3E79BA0 on vertex vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,169 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:27,170 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {BE61D95195A2460A98450799B3E79BA0=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:27,170 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] parallelism set to 1
2016-05-18 00:57:27,171 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:27,172 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E004C2E204794287A7B15EF0E9697CEC, vertexId=vertex_1463533046973_0001_1_00, initRequestedTime=1463533047151, initedTime=1463533047172, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:27,172 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] with distanceFromRoot: 0
2016-05-18 00:57:27,173 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:27,173 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533046973_0001_1_00, startRequestedTime=1463533047155, startedTime=1463533047173
2016-05-18 00:57:27,174 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533046973_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:27,174 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:27,175 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533046973_0001_1_00 for vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8] numStartedSources: 1 numSources: 1
2016-05-18 00:57:27,175 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8] with distanceFromRoot: 1
2016-05-18 00:57:27,175 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E004C2E204794287A7B15EF0E9697CEC
2016-05-18 00:57:27,176 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,177 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533046973_0001_1_01, startRequestedTime=1463533047175, startedTime=1463533047176
2016-05-18 00:57:27,177 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:27,178 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:TASK_STARTED]: vertexName=E004C2E204794287A7B15EF0E9697CEC, taskId=task_1463533046973_0001_1_00_000000, scheduledTime=1463533047178, launchTime=1463533047178
2016-05-18 00:57:27,178 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,181 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:27,181 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: ABA91BBB17DF456AB4B966F6327F8FB8 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:27,183 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E004C2E204794287A7B15EF0E9697CEC in vertex: ABA91BBB17DF456AB4B966F6327F8FB8 numBipartiteSourceTasks: 1
2016-05-18 00:57:27,184 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533046973_0001, with dag id: dag_1463533046973_0001_1
2016-05-18 00:57:27,186 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533046973_0001_00_000001, launchTime=1463533047185
2016-05-18 00:57:27,187 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533046973_0001_00_000001
2016-05-18 00:57:27,372 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533046973_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533046973_0001_1_00_000000_0
2016-05-18 00:57:27 Starting to run new task attempt: attempt_1463533046973_0001_1_00_000000_0
2016-05-18 00:57:27,373 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533046973_0001_1_00_000000_0
2016-05-18 00:57:27,373 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533046973_0001_1_00_000000_0] started. Is using containerId: [container_1463533046973_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:27,374 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E004C2E204794287A7B15EF0E9697CEC, taskAttemptId=attempt_1463533046973_0001_1_00_000000_0, startTime=1463533047373, containerId=container_1463533046973_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:27,374 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:27,374 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:27,374 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030] every/(1/1) ...blyplatform/-1_UNKNOWN_-1, VertexName: E004C2E204794287A7B15EF0E9697CEC, VertexParallelism: 1, TaskAttemptID:attempt_1463533046973_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BE61D95195A2460A98450799B3E79BA0, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=ABA91BBB17DF456AB4B966F6327F8FB8, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:27,375 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:27,375 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533046973_0001_1_00_000000_0
2016-05-18 00:57:27,377 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533046973_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:27,378 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533046973_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:27,381 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:27,385 INFO input.MRInput (MRInput.java:initialize(440)) - BE61D95195A2460A98450799B3E79BA0 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:27,386 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: BE61D95195A2460A98450799B3E79BA0
2016-05-18 00:57:27,390 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E004C2E204794287A7B15EF0E9697CEC, ordinal: 0
2016-05-18 00:57:27,390 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E004C2E204794287A7B15EF0E9697CEC, mem on start (mb), free: 339, total: 364, max: 672
2016-05-18 00:57:27,390 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:27,390 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:27,390 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:27,390 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:27,390 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BE61D95195A2460A98450799B3E79BA0:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [ABA91BBB17DF456AB4B966F6327F8FB8:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:27,391 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:27,391 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:27,391 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:27,392 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BE61D95195A2460A98450799B3E79BA0:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [ABA91BBB17DF456AB4B966F6327F8FB8:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:27,392 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:27,392 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BE61D95195A2460A98450799B3E79BA0 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:27,393 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:27,393 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BE61D95195A2460A98450799B3E79BA0
2016-05-18 00:57:27,393 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:27,394 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:27,394 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533046973_0001_1_00_000000_0
2016-05-18 00:57:27,398 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:27,399 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: BE61D95195A2460A98450799B3E79BA0
2016-05-18 00:57:27,399 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 9556F0956BBA4DD2A915909D29460A28
2016-05-18 00:57:27,399 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 9556F0956BBA4DD2A915909D29460A28
2016-05-18 00:57:27,399 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:27,400 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - BE61D95195A2460A98450799B3E79BA0 initialized RecordReader from event
2016-05-18 00:57:27,401 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:27,404 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:27,404 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - ABA91BBB17DF456AB4B966F6327F8FB8 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6c00b29f, comparator=cascading.tuple.hadoop.util.TupleComparator@243f06e9, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:27,404 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for ABA91BBB17DF456AB4B966F6327F8FB8: , UsingHashComparator=false
2016-05-18 00:57:27,417 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:27,417 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:27,417 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for ABA91BBB17DF456AB4B966F6327F8FB8: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:27,417 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - ABA91BBB17DF456AB4B966F6327F8FB8: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:27,418 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:27,418 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E004C2E204794287A7B15EF0E9697CEC, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:27,423 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533046973_0001_1_00_000000_0
2016-05-18 00:57:27,424 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E004C2E204794287A7B15EF0E9697CEC, mem on close (mb), free: 237, total: 364, max: 672
2016-05-18 00:57:27,424 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - ABA91BBB17DF456AB4B966F6327F8FB8: Starting flush of map output
2016-05-18 00:57:27,424 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - ABA91BBB17DF456AB4B966F6327F8FB8: Span0.length = 20, perItem = 6
2016-05-18 00:57:27,424 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - ABA91BBB17DF456AB4B966F6327F8FB8: done sorting span=0, length=20, time=0
2016-05-18 00:57:27,425 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - ABA91BBB17DF456AB4B966F6327F8FB8: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:27,426 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - ABA91BBB17DF456AB4B966F6327F8FB8: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001_wd/localmode-local-dir/output/attempt_1463533046973_0001_1_00_000000_0_10032_0/file.out
2016-05-18 00:57:27,430 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:27 Completed running task attempt: attempt_1463533046973_0001_1_00_000000_0
2016-05-18 00:57:27,430 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533046973_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:27,430 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533046973_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:27,431 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533046973_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533047419, Process_Duration=4, Process_End_Time=1463533047423, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:27,432 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533046973_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:27,433 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:27,433 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:27,433 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:27,433 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533046973_0001_00_000001
2016-05-18 00:57:27,433 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E004C2E204794287A7B15EF0E9697CEC, taskAttemptId=attempt_1463533046973_0001_1_00_000000_0, creationTime=1463533047178, allocationTime=1463533047183, startTime=1463533047373, finishTime=1463533047433, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:27,435 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533046973_0001_1_00_000000_0
2016-05-18 00:57:27,436 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:TASK_FINISHED]: vertexName=E004C2E204794287A7B15EF0E9697CEC, taskId=task_1463533046973_0001_1_00_000000, startTime=1463533047373, finishTime=1463533047436, timeTaken=63, status=SUCCEEDED, successfulAttemptID=attempt_1463533046973_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533047419, Process_Duration=4, Process_End_Time=1463533047423, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:27,437 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:27,437 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,438 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:VERTEX_FINISHED]: vertexName=E004C2E204794287A7B15EF0E9697CEC, vertexId=vertex_1463533046973_0001_1_00, initRequestedTime=1463533047151, initedTime=1463533047172, startRequestedTime=1463533047155, startedTime=1463533047173, finishTime=1463533047437, timeTaken=264, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533047419, Process_Duration=4, Process_End_Time=1463533047423, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533047373, firstTasksToStart=[ task_1463533046973_0001_1_00_000000 ], lastTaskFinishTime=1463533047433, lastTasksToFinish=[ task_1463533046973_0001_1_00_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533046973_0001_1_00_000000 ], longestDurationTasks=[ task_1463533046973_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:27,438 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:27,439 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533046973_0001_00_000001, stoppedTime=1463533047439, exitStatus=0
2016-05-18 00:57:27,439 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8] attempt: attempt_1463533046973_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:27,439 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:27,439 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: ABA91BBB17DF456AB4B966F6327F8FB8
2016-05-18 00:57:27,440 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E004C2E204794287A7B15EF0E9697CEC, destinationVertexName=ABA91BBB17DF456AB4B966F6327F8FB8. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:27,440 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]
2016-05-18 00:57:27,440 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:27,441 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533046973_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:27,441 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:TASK_STARTED]: vertexName=ABA91BBB17DF456AB4B966F6327F8FB8, taskId=task_1463533046973_0001_1_01_000000, scheduledTime=1463533047441, launchTime=1463533047441
2016-05-18 00:57:27,442 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]
2016-05-18 00:57:27,443 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533046973_0001_00_000002, launchTime=1463533047443
2016-05-18 00:57:27,444 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533046973_0001_00_000002
2016-05-18 00:57:27,445 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533046973_0001_1_01_000000_0] started. Is using containerId: [container_1463533046973_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:27,445 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ABA91BBB17DF456AB4B966F6327F8FB8, taskAttemptId=attempt_1463533046973_0001_1_01_000000_0, startTime=1463533047444, containerId=container_1463533046973_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:27,445 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533046973_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533046973_0001_1_01_000000_0
2016-05-18 00:57:27 Starting to run new task attempt: attempt_1463533046973_0001_1_01_000000_0
2016-05-18 00:57:27,446 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533046973_0001_1_01_000000_0
2016-05-18 00:57:27,446 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:27,447 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:27,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030] every/(1/1) ...blyplatform/-1_UNKNOWN_-1, VertexName: ABA91BBB17DF456AB4B966F6327F8FB8, VertexParallelism: 1, TaskAttemptID:attempt_1463533046973_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E004C2E204794287A7B15EF0E9697CEC, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=6AD48C4884894F16AC2FB1B6A591DA0A, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:27,447 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533046973_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:27,447 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533046973_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:27,447 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:27,447 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533046973_0001_00_000001, Exiting
2016-05-18 00:57:27,448 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533046973_0001_00_000001
2016-05-18 00:57:27,448 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533046973_0001_00_000001
2016-05-18 00:57:27,448 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533046973_0001_1_01_000000_0
2016-05-18 00:57:27,450 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533046973_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]
2016-05-18 00:57:27,451 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533046973_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:27,453 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533046973_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:27,453 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533046973_0001_00_000001 completed successfully
2016-05-18 00:57:27,455 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:27,453 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:27,463 INFO output.MROutput (MROutput.java:initialize(421)) - 6AD48C4884894F16AC2FB1B6A591DA0A: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:27,466 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ABA91BBB17DF456AB4B966F6327F8FB8, ordinal: 1
2016-05-18 00:57:27,466 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ABA91BBB17DF456AB4B966F6327F8FB8, mem on start (mb), free: 232, total: 364, max: 672
2016-05-18 00:57:27,466 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:27,466 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:27,466 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:27,467 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:27,467 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6AD48C4884894F16AC2FB1B6A591DA0A:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [E004C2E204794287A7B15EF0E9697CEC:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:27,467 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:27,467 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:27,468 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:27,468 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6AD48C4884894F16AC2FB1B6A591DA0A:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [E004C2E204794287A7B15EF0E9697CEC:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:27,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:27,468 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E004C2E204794287A7B15EF0E9697CEC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:27,469 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:27,469 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:27,469 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E004C2E204794287A7B15EF0E9697CEC: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:27,469 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E004C2E204794287A7B15EF0E9697CEC: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:27,470 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E004C2E204794287A7B15EF0E9697CEC with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:27,471 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E004C2E204794287A7B15EF0E9697CEC
2016-05-18 00:57:27,472 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:27,472 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533046973_0001_1_01_000000_0
2016-05-18 00:57:27,472 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E004C2E204794287A7B15EF0E9697CEC: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:27,474 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533046973_0001_1_00_000000_0_10032} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533047474, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:27,475 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E004C2E204794287A7B15EF0E9697CEC
2016-05-18 00:57:27,475 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:27,475 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E004C2E204794287A7B15EF0E9697CEC, wasInterrupted=false
2016-05-18 00:57:27,476 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:27,476 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:27,477 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 9556F0956BBA4DD2A915909D29460A28
2016-05-18 00:57:27,477 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_-1"], id: 6AD48C4884894F16AC2FB1B6A591DA0A
2016-05-18 00:57:27,478 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_-1"]
2016-05-18 00:57:27,478 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 9556F0956BBA4DD2A915909D29460A28, for 1 inputs
2016-05-18 00:57:27,478 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:27,480 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:27,480 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001_wd/localmode-local-dir/output/attempt_1463533046973_0001_1_00_000000_0_10032/file.out, len=181, isLocal=true
2016-05-18 00:57:27,480 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:27,480 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:27,481 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:27,481 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:27,481 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E004C2E204794287A7B15EF0E9697CEC
2016-05-18 00:57:27,481 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E004C2E204794287A7B15EF0E9697CEC: Shuffle Runner thread complete
2016-05-18 00:57:27,482 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ABA91BBB17DF456AB4B966F6327F8FB8, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:57:27,483 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:27,483 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E004C2E204794287A7B15EF0E9697CEC: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:27,490 INFO output.MROutput (MROutput.java:close(526)) - 6AD48C4884894F16AC2FB1B6A591DA0A closed
2016-05-18 00:57:27,492 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533046973_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:27,493 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533046973_0001_1_01_000000_0
2016-05-18 00:57:27,493 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ABA91BBB17DF456AB4B966F6327F8FB8, mem on close (mb), free: 229, total: 364, max: 672
2016-05-18 00:57:27,494 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E004C2E204794287A7B15EF0E9697CEC
2016-05-18 00:57:27,494 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E004C2E204794287A7B15EF0E9697CEC: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:27,495 INFO output.MROutput (MROutput.java:close(526)) - 6AD48C4884894F16AC2FB1B6A591DA0A closed
2016-05-18 00:57:27,495 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:27 Completed running task attempt: attempt_1463533046973_0001_1_01_000000_0
2016-05-18 00:57:27,497 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533046973_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:27,497 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533046973_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:27,498 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533046973_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533047482, Process_Duration=11, Process_End_Time=1463533047493, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:27,499 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533046973_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:27,499 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:27,500 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:27,500 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:27,500 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ABA91BBB17DF456AB4B966F6327F8FB8, taskAttemptId=attempt_1463533046973_0001_1_01_000000_0, creationTime=1463533047441, allocationTime=1463533047442, startTime=1463533047444, finishTime=1463533047500, timeTaken=56, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:27,500 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533046973_0001_1_01_000000_0
2016-05-18 00:57:27,501 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533046973_0001_00_000002
2016-05-18 00:57:27,501 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:TASK_FINISHED]: vertexName=ABA91BBB17DF456AB4B966F6327F8FB8, taskId=task_1463533046973_0001_1_01_000000, startTime=1463533047444, finishTime=1463533047500, timeTaken=56, status=SUCCEEDED, successfulAttemptID=attempt_1463533046973_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533047482, Process_Duration=11, Process_End_Time=1463533047493, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:27,502 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:27,502 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8]
2016-05-18 00:57:27,503 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:VERTEX_FINISHED]: vertexName=ABA91BBB17DF456AB4B966F6327F8FB8, vertexId=vertex_1463533046973_0001_1_01, initRequestedTime=1463533047155, initedTime=1463533047159, startRequestedTime=1463533047175, startedTime=1463533047176, finishTime=1463533047502, timeTaken=326, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533047482, Process_Duration=11, Process_End_Time=1463533047493, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533047444, firstTasksToStart=[ task_1463533046973_0001_1_01_000000 ], lastTaskFinishTime=1463533047500, lastTasksToFinish=[ task_1463533046973_0001_1_01_000000 ], minTaskDuration=56, maxTaskDuration=56, avgTaskDuration=56.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533046973_0001_1_01_000000 ], longestDurationTasks=[ task_1463533046973_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:27,504 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:27,504 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533046973_0001_00_000002, stoppedTime=1463533047504, exitStatus=0
2016-05-18 00:57:27,504 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:27,504 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:27,505 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533046973_0001_1_00 [E004C2E204794287A7B15EF0E9697CEC]
2016-05-18 00:57:27,505 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533046973_0001_1
2016-05-18 00:57:27,505 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533046973_0001_1
2016-05-18 00:57:27,506 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533046973_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:27,506 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 6AD48C4884894F16AC2FB1B6A591DA0A for vertex: vertex_1463533046973_0001_1_01 [ABA91BBB17DF456AB4B966F6327F8FB8], outputName: 6AD48C4884894F16AC2FB1B6A591DA0A
2016-05-18 00:57:27,506 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533046973_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:27,512 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533046973_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:27,512 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533046973_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:27,512 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533046973_0001_00_000002, Exiting
2016-05-18 00:57:27,513 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533046973_0001_00_000002
2016-05-18 00:57:27,513 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533046973_0001_00_000002
2016-05-18 00:57:27,513 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533046973_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:27,513 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533046973_0001_00_000002 completed successfully
2016-05-18 00:57:27,513 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:6AD48C4884894F16AC2FB1B6A591DA0A of vertex/vertexGroup:ABA91BBB17DF456AB4B966F6327F8FB8 isVertexGroupOutput:false
2016-05-18 00:57:27,514 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:27,516 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533046973_0001_1, queueSize=0
2016-05-18 00:57:27,517 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533046973_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533046973_0001_1, startTime=1463533047150, finishTime=1463533047514, timeTaken=364, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=780, AM_GC_TIME_MILLIS=184, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066094901, Process_Duration=15, Process_End_Time=2927066094916, Read_Duration=6, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:27,517 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533046973_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:27,517 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533046973_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:27 Completed Dag: dag_1463533046973_0001_1
2016-05-18 00:57:27,517 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533046973_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:27,517 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533046973_0001_1_post
2016-05-18 00:57:27,518 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:27,518 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:27,518 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:27,518 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030] every/(1/1) ...blyplatform/-1_UNKNOWN_-1, with id=dag_1463533046973_0001_1
2016-05-18 00:57:27,518 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030] every/(1/1) ...blyplatform/-1_UNKNOWN_-1, with id=dag_1463533046973_0001_1
2016-05-18 00:57:27,526 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...blyplatform/-1_UNKNOWN_-1, applicationId=application_1463533046973_0001
2016-05-18 00:57:27,528 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:27,528 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:27,528 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:27,529 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:27,529 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:27,529 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:27,530 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:27,530 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:27,530 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:27,530 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:27,531 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:27,531 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:27,532 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:27,532 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:27,536 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4E15652EA5584A7EB9F05134409CEA97/EBB53F4F9A00428AA7044B80A27B0030/.tez/application_1463533046973_0001
2016-05-18 00:57:27,536 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_-1/_temporary
2016-05-18 00:57:27,537 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.587
2016-05-18 00:57:27,612 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_UNKNOWN_-1
2016-05-18 00:57:27,637 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:27,637 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:27,674 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.008
2016-05-18 00:57:27,677 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.011
2016-05-18 00:57:27,678 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:27,678 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:27,678 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_UNKNOWN_'field'
2016-05-18 00:57:27,703 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:27,704 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:27,756 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.024
2016-05-18 00:57:27,766 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.034
2016-05-18 00:57:27,766 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:27,766 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:27,767 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:27,784 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:27,791 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_RESULTS
2016-05-18 00:57:27,847 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:27,847 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:27,848 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_RESULTS"]
2016-05-18 00:57:27,848 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:27,848 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:27,849 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:27,849 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...atform/-1_UNKNOWN_RESULTS
2016-05-18 00:57:27,849 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:27,870 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:27,875 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:27,875 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:27,875 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:27,885 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001 doesn't exist and is created
2016-05-18 00:57:27,899 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:27,900 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:27,901 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001_wd
2016-05-18 00:57:27,911 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533047875_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:27,912 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:27,912 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:27,914 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:27,914 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:27,914 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:27,914 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:27,915 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:27,917 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:27,917 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:27,917 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:27,917 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:27,918 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:27,918 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:27,918 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:27,918 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:27,919 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:27,919 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:27,919 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:27,920 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:27,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:27,922 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:27,922 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:27,922 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001_wd/localmode-log-dir
2016-05-18 00:57:27,923 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001_wd/localmode-log-dir/history.txt.appattempt_1463533047875_0001_000000, maxErrors=10
2016-05-18 00:57:27,923 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:27,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533047875_0001_000000, appSubmitTime=1463533047911, launchTime=1463533047911
2016-05-18 00:57:27,931 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:27,937 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:54201
2016-05-18 00:57:27,942 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533047875
2016-05-18 00:57:27,944 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533047875_0001_000000, startTime=1463533047944
2016-05-18 00:57:27,944 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:28,000 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:28,000 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:28,000 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...atform/-1_UNKNOWN_RESULTS, applicationId=application_1463533047875_0001, dagName=[0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95] every/(1/1) ...atform/-1_UNKNOWN_RESULTS
2016-05-18 00:57:28,020 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95] every/(1/1) ...atform/-1_UNKNOWN_RESULTS
2016-05-18 00:57:28,020 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:28,021 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533047875_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001_wd/localmode-log-dir/dag_1463533047875_0001_1.dot
2016-05-18 00:57:28,022 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001_wd/localmode-log-dir/dag_1463533047875_0001_1-tez-dag.pb.txt
2016-05-18 00:57:28,029 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533047875_0001_1
2016-05-18 00:57:28,029 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,029 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95] every/(1/1) ...atform/-1_UNKNOWN_RESULTS
2016-05-18 00:57:28 Running Dag: dag_1463533047875_0001_1
2016-05-18 00:57:28,039 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533047875_0001_1, submitTime=1463533048020
2016-05-18 00:57:28,040 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001/tez-conf.pb" } size: 36913 timestamp: 1463533047000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:28,040 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,041 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:28,042 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001/tez-conf.pb" } size: 36913 timestamp: 1463533047000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:28,042 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]
2016-05-18 00:57:28,042 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:28,043 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:28,043 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533047875_0001_1, initTime=1463533048043
2016-05-18 00:57:28,044 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533047875_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:28,044 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:28,045 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:DAG_STARTED]: dagID=dag_1463533047875_0001_1, startTime=1463533048044
2016-05-18 00:57:28,045 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533047875_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:28,045 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D02E64C0AF4B4E2BB0ECF556D8744258 : {CAA56D960CBB4044BB31E3EBCC4882D9={InputName=CAA56D960CBB4044BB31E3EBCC4882D9}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:28,045 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: CAA56D960CBB4044BB31E3EBCC4882D9, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:28,045 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,045 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,045 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,046 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,046 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:28,047 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]
2016-05-18 00:57:28,046 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: CAA56D960CBB4044BB31E3EBCC4882D9 on vertex vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,048 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input CAA56D960CBB4044BB31E3EBCC4882D9 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:28,050 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:28,050 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]
2016-05-18 00:57:28,051 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]
2016-05-18 00:57:28,051 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533047875_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:28,050 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:28,051 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E], numAdditionalOutputs=1
2016-05-18 00:57:28,052 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=305A0FEF7F9E44ED96DDE1627C870214, vertex=vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:28,053 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:28,053 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:28,053 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:28,054 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: CAA56D960CBB4044BB31E3EBCC4882D9 on vertex vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,054 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...atform/-1_UNKNOWN_RESULTS, applicationId=application_1463533047875_0001, dagName=[0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95] every/(1/1) ...atform/-1_UNKNOWN_RESULTS
2016-05-18 00:57:28,054 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for CF42967C11B94C59AD48844062EE012E:305A0FEF7F9E44ED96DDE1627C870214 using oldmapred API
2016-05-18 00:57:28,055 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=305A0FEF7F9E44ED96DDE1627C870214, vertexName=CF42967C11B94C59AD48844062EE012E, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:28,056 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CF42967C11B94C59AD48844062EE012E, vertexId=vertex_1463533047875_0001_1_01, initRequestedTime=1463533048047, initedTime=1463533048051, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:28,056 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:28,058 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:28,059 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {CAA56D960CBB4044BB31E3EBCC4882D9=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:28,059 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] parallelism set to 1
2016-05-18 00:57:28,060 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:28,060 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D02E64C0AF4B4E2BB0ECF556D8744258, vertexId=vertex_1463533047875_0001_1_00, initRequestedTime=1463533048045, initedTime=1463533048060, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:28,061 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] with distanceFromRoot: 0
2016-05-18 00:57:28,061 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:28,061 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533047875_0001_1_00, startRequestedTime=1463533048047, startedTime=1463533048061
2016-05-18 00:57:28,062 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533047875_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:28,062 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:28,062 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533047875_0001_1_00 for vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E] numStartedSources: 1 numSources: 1
2016-05-18 00:57:28,063 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E] with distanceFromRoot: 1
2016-05-18 00:57:28,062 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D02E64C0AF4B4E2BB0ECF556D8744258
2016-05-18 00:57:28,063 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,064 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533047875_0001_1_01, startRequestedTime=1463533048062, startedTime=1463533048063
2016-05-18 00:57:28,065 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:28,067 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: CF42967C11B94C59AD48844062EE012E with 0 source tasks and 1 pending tasks
2016-05-18 00:57:28,067 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:TASK_STARTED]: vertexName=D02E64C0AF4B4E2BB0ECF556D8744258, taskId=task_1463533047875_0001_1_00_000000, scheduledTime=1463533048067, launchTime=1463533048067
2016-05-18 00:57:28,068 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,068 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D02E64C0AF4B4E2BB0ECF556D8744258 in vertex: CF42967C11B94C59AD48844062EE012E numBipartiteSourceTasks: 1
2016-05-18 00:57:28,069 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:28,071 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533047875_0001_00_000001, launchTime=1463533048070
2016-05-18 00:57:28,071 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533047875_0001_00_000001
2016-05-18 00:57:28,072 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533047875_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533047875_0001_1_00_000000_0
2016-05-18 00:57:28,072 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533047875_0001_1_00_000000_0] started. Is using containerId: [container_1463533047875_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:28 Starting to run new task attempt: attempt_1463533047875_0001_1_00_000000_0
2016-05-18 00:57:28,073 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533047875_0001_1_00_000000_0
2016-05-18 00:57:28,073 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,073 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:28,073 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95] every/(1/1) ...atform/-1_UNKNOWN_RESULTS, VertexName: D02E64C0AF4B4E2BB0ECF556D8744258, VertexParallelism: 1, TaskAttemptID:attempt_1463533047875_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CAA56D960CBB4044BB31E3EBCC4882D9, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CF42967C11B94C59AD48844062EE012E, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:28,073 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:28,074 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D02E64C0AF4B4E2BB0ECF556D8744258, taskAttemptId=attempt_1463533047875_0001_1_00_000000_0, startTime=1463533048072, containerId=container_1463533047875_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:28,075 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533047875_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:28,076 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533047875_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:28,076 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533047875_0001_1_00_000000_0
2016-05-18 00:57:28,080 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:28,084 INFO input.MRInput (MRInput.java:initialize(440)) - CAA56D960CBB4044BB31E3EBCC4882D9 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:28,084 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: CAA56D960CBB4044BB31E3EBCC4882D9
2016-05-18 00:57:28,085 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533047875_0001, with dag id: dag_1463533047875_0001_1
2016-05-18 00:57:28,092 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D02E64C0AF4B4E2BB0ECF556D8744258, ordinal: 0
2016-05-18 00:57:28,092 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D02E64C0AF4B4E2BB0ECF556D8744258, mem on start (mb), free: 316, total: 365, max: 672
2016-05-18 00:57:28,092 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:28,092 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:28,092 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:28,092 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:28,093 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CAA56D960CBB4044BB31E3EBCC4882D9:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [CF42967C11B94C59AD48844062EE012E:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:28,093 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:28,093 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:28,094 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:28,094 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CAA56D960CBB4044BB31E3EBCC4882D9:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [CF42967C11B94C59AD48844062EE012E:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:28,094 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:28,094 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CAA56D960CBB4044BB31E3EBCC4882D9 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:28,095 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:28,095 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:28,094 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CAA56D960CBB4044BB31E3EBCC4882D9
2016-05-18 00:57:28,096 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:28,096 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533047875_0001_1_00_000000_0
2016-05-18 00:57:28,098 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:28,098 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - CAA56D960CBB4044BB31E3EBCC4882D9 initialized RecordReader from event
2016-05-18 00:57:28,100 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:28,100 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: CAA56D960CBB4044BB31E3EBCC4882D9
2016-05-18 00:57:28,100 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 8642DDE513B74253AA3DEBB54F773441
2016-05-18 00:57:28,101 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 8642DDE513B74253AA3DEBB54F773441
2016-05-18 00:57:28,102 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:28,106 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:28,106 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CF42967C11B94C59AD48844062EE012E using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@5adb45f2, comparator=cascading.tuple.hadoop.util.TupleComparator@71cd4177, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:28,106 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CF42967C11B94C59AD48844062EE012E: , UsingHashComparator=false
2016-05-18 00:57:28,120 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:28,120 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:28,120 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CF42967C11B94C59AD48844062EE012E: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:28,120 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CF42967C11B94C59AD48844062EE012E: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:28,120 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:28,121 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D02E64C0AF4B4E2BB0ECF556D8744258, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:28,123 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533047875_0001_1_00_000000_0
2016-05-18 00:57:28,123 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D02E64C0AF4B4E2BB0ECF556D8744258, mem on close (mb), free: 214, total: 365, max: 672
2016-05-18 00:57:28,124 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CF42967C11B94C59AD48844062EE012E: Starting flush of map output
2016-05-18 00:57:28,124 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CF42967C11B94C59AD48844062EE012E: Span0.length = 20, perItem = 6
2016-05-18 00:57:28,124 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CF42967C11B94C59AD48844062EE012E: done sorting span=0, length=20, time=0
2016-05-18 00:57:28,125 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CF42967C11B94C59AD48844062EE012E: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:28,126 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CF42967C11B94C59AD48844062EE012E: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001_wd/localmode-local-dir/output/attempt_1463533047875_0001_1_00_000000_0_10039_0/file.out
2016-05-18 00:57:28,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:28 Completed running task attempt: attempt_1463533047875_0001_1_00_000000_0
2016-05-18 00:57:28,131 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533047875_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:28,132 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533047875_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:28,132 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533047875_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533048121, Process_Duration=2, Process_End_Time=1463533048123, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:28,133 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533047875_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:28,134 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:28,134 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:28,134 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,134 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D02E64C0AF4B4E2BB0ECF556D8744258, taskAttemptId=attempt_1463533047875_0001_1_00_000000_0, creationTime=1463533048068, allocationTime=1463533048070, startTime=1463533048072, finishTime=1463533048134, timeTaken=62, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:28,135 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533047875_0001_1_00_000000_0
2016-05-18 00:57:28,135 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:TASK_FINISHED]: vertexName=D02E64C0AF4B4E2BB0ECF556D8744258, taskId=task_1463533047875_0001_1_00_000000, startTime=1463533048072, finishTime=1463533048135, timeTaken=63, status=SUCCEEDED, successfulAttemptID=attempt_1463533047875_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533048121, Process_Duration=2, Process_End_Time=1463533048123, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:28,136 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533047875_0001_00_000001
2016-05-18 00:57:28,136 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:28,136 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,137 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:VERTEX_FINISHED]: vertexName=D02E64C0AF4B4E2BB0ECF556D8744258, vertexId=vertex_1463533047875_0001_1_00, initRequestedTime=1463533048045, initedTime=1463533048060, startRequestedTime=1463533048047, startedTime=1463533048061, finishTime=1463533048136, timeTaken=75, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533048121, Process_Duration=2, Process_End_Time=1463533048123, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533048072, firstTasksToStart=[ task_1463533047875_0001_1_00_000000 ], lastTaskFinishTime=1463533048134, lastTasksToFinish=[ task_1463533047875_0001_1_00_000000 ], minTaskDuration=62, maxTaskDuration=62, avgTaskDuration=62.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533047875_0001_1_00_000000 ], longestDurationTasks=[ task_1463533047875_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:28,138 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:28,138 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533047875_0001_00_000001, stoppedTime=1463533048138, exitStatus=0
2016-05-18 00:57:28,138 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E] attempt: attempt_1463533047875_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:28,139 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:28,139 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:28,139 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533047875_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:28,140 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CF42967C11B94C59AD48844062EE012E
2016-05-18 00:57:28,140 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D02E64C0AF4B4E2BB0ECF556D8744258, destinationVertexName=CF42967C11B94C59AD48844062EE012E. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:28,140 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]
2016-05-18 00:57:28,141 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:TASK_STARTED]: vertexName=CF42967C11B94C59AD48844062EE012E, taskId=task_1463533047875_0001_1_01_000000, scheduledTime=1463533048141, launchTime=1463533048141
2016-05-18 00:57:28,141 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]
2016-05-18 00:57:28,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533047875_0001_00_000002, launchTime=1463533048144
2016-05-18 00:57:28,145 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533047875_0001_00_000002
2016-05-18 00:57:28,146 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533047875_0001_1_01_000000_0] started. Is using containerId: [container_1463533047875_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:28,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CF42967C11B94C59AD48844062EE012E, taskAttemptId=attempt_1463533047875_0001_1_01_000000_0, startTime=1463533048146, containerId=container_1463533047875_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:28,147 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533047875_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:28,147 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533047875_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:28,147 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533047875_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533047875_0001_1_01_000000_0
2016-05-18 00:57:28,147 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533047875_0001_00_000001, Exiting
2016-05-18 00:57:28,147 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533047875_0001_00_000001
2016-05-18 00:57:28,147 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533047875_0001_00_000001
2016-05-18 00:57:28 Starting to run new task attempt: attempt_1463533047875_0001_1_01_000000_0
2016-05-18 00:57:28,148 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533047875_0001_1_01_000000_0
2016-05-18 00:57:28,148 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,148 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:28,148 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533047875_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:28,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95] every/(1/1) ...atform/-1_UNKNOWN_RESULTS, VertexName: CF42967C11B94C59AD48844062EE012E, VertexParallelism: 1, TaskAttemptID:attempt_1463533047875_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D02E64C0AF4B4E2BB0ECF556D8744258, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=305A0FEF7F9E44ED96DDE1627C870214, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:28,148 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533047875_0001_00_000001 completed successfully
2016-05-18 00:57:28,148 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:28,149 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533047875_0001_1_01_000000_0
2016-05-18 00:57:28,151 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533047875_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]
2016-05-18 00:57:28,151 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533047875_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:28,154 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:28,156 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:28,163 INFO output.MROutput (MROutput.java:initialize(421)) - 305A0FEF7F9E44ED96DDE1627C870214: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:28,165 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CF42967C11B94C59AD48844062EE012E, ordinal: 1
2016-05-18 00:57:28,165 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF42967C11B94C59AD48844062EE012E, mem on start (mb), free: 209, total: 365, max: 672
2016-05-18 00:57:28,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:28,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:28,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:28,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:28,165 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[305A0FEF7F9E44ED96DDE1627C870214:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [D02E64C0AF4B4E2BB0ECF556D8744258:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:28,166 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:28,166 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:28,166 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:28,167 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[305A0FEF7F9E44ED96DDE1627C870214:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [D02E64C0AF4B4E2BB0ECF556D8744258:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:28,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:28,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D02E64C0AF4B4E2BB0ECF556D8744258 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:28,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:28,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:28,171 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D02E64C0AF4B4E2BB0ECF556D8744258: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:28,171 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D02E64C0AF4B4E2BB0ECF556D8744258: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:28,172 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D02E64C0AF4B4E2BB0ECF556D8744258 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:28,173 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D02E64C0AF4B4E2BB0ECF556D8744258
2016-05-18 00:57:28,175 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:28,176 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533047875_0001_1_01_000000_0
2016-05-18 00:57:28,176 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D02E64C0AF4B4E2BB0ECF556D8744258: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:28,179 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533047875_0001_1_00_000000_0_10039} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533048179, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:28,179 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:28,179 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D02E64C0AF4B4E2BB0ECF556D8744258
2016-05-18 00:57:28,179 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:28,179 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 8642DDE513B74253AA3DEBB54F773441
2016-05-18 00:57:28,180 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D02E64C0AF4B4E2BB0ECF556D8744258, wasInterrupted=false
2016-05-18 00:57:28,180 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_RESULTS"], id: 305A0FEF7F9E44ED96DDE1627C870214
2016-05-18 00:57:28,181 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_RESULTS"]
2016-05-18 00:57:28,181 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:28,181 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 8642DDE513B74253AA3DEBB54F773441, for 1 inputs
2016-05-18 00:57:28,181 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:28,183 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:28,184 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001_wd/localmode-local-dir/output/attempt_1463533047875_0001_1_00_000000_0_10039/file.out, len=181, isLocal=true
2016-05-18 00:57:28,184 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:28,185 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:28,185 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:28,185 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:28,185 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D02E64C0AF4B4E2BB0ECF556D8744258
2016-05-18 00:57:28,185 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D02E64C0AF4B4E2BB0ECF556D8744258: Shuffle Runner thread complete
2016-05-18 00:57:28,185 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CF42967C11B94C59AD48844062EE012E, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:28,186 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:28,187 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D02E64C0AF4B4E2BB0ECF556D8744258: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:28,197 INFO output.MROutput (MROutput.java:close(526)) - 305A0FEF7F9E44ED96DDE1627C870214 closed
2016-05-18 00:57:28,198 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533047875_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:28,199 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533047875_0001_1_01_000000_0
2016-05-18 00:57:28,199 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF42967C11B94C59AD48844062EE012E, mem on close (mb), free: 206, total: 365, max: 672
2016-05-18 00:57:28,199 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D02E64C0AF4B4E2BB0ECF556D8744258
2016-05-18 00:57:28,199 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D02E64C0AF4B4E2BB0ECF556D8744258: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:28,200 INFO output.MROutput (MROutput.java:close(526)) - 305A0FEF7F9E44ED96DDE1627C870214 closed
2016-05-18 00:57:28,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:28 Completed running task attempt: attempt_1463533047875_0001_1_01_000000_0
2016-05-18 00:57:28,201 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533047875_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:28,201 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533047875_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:28,201 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533047875_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533048186, Process_Duration=13, Process_End_Time=1463533048199, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:28,202 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533047875_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:28,202 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:28,203 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CF42967C11B94C59AD48844062EE012E, taskAttemptId=attempt_1463533047875_0001_1_01_000000_0, creationTime=1463533048141, allocationTime=1463533048143, startTime=1463533048146, finishTime=1463533048202, timeTaken=56, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:28,203 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:28,203 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,203 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533047875_0001_1_01_000000_0
2016-05-18 00:57:28,203 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533047875_0001_00_000002
2016-05-18 00:57:28,204 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:TASK_FINISHED]: vertexName=CF42967C11B94C59AD48844062EE012E, taskId=task_1463533047875_0001_1_01_000000, startTime=1463533048146, finishTime=1463533048203, timeTaken=57, status=SUCCEEDED, successfulAttemptID=attempt_1463533047875_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533048186, Process_Duration=13, Process_End_Time=1463533048199, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:28,204 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:28,204 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E]
2016-05-18 00:57:28,206 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:VERTEX_FINISHED]: vertexName=CF42967C11B94C59AD48844062EE012E, vertexId=vertex_1463533047875_0001_1_01, initRequestedTime=1463533048047, initedTime=1463533048051, startRequestedTime=1463533048062, startedTime=1463533048063, finishTime=1463533048205, timeTaken=142, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533048186, Process_Duration=13, Process_End_Time=1463533048199, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533048146, firstTasksToStart=[ task_1463533047875_0001_1_01_000000 ], lastTaskFinishTime=1463533048202, lastTasksToFinish=[ task_1463533047875_0001_1_01_000000 ], minTaskDuration=56, maxTaskDuration=56, avgTaskDuration=56.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533047875_0001_1_01_000000 ], longestDurationTasks=[ task_1463533047875_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:28,206 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:28,206 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533047875_0001_00_000002, stoppedTime=1463533048206, exitStatus=0
2016-05-18 00:57:28,207 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:28,207 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:28,207 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533047875_0001_1_00 [D02E64C0AF4B4E2BB0ECF556D8744258]
2016-05-18 00:57:28,207 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533047875_0001_1
2016-05-18 00:57:28,207 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533047875_0001_1
2016-05-18 00:57:28,208 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 305A0FEF7F9E44ED96DDE1627C870214 for vertex: vertex_1463533047875_0001_1_01 [CF42967C11B94C59AD48844062EE012E], outputName: 305A0FEF7F9E44ED96DDE1627C870214
2016-05-18 00:57:28,209 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533047875_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:28,210 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533047875_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:28,214 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533047875_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:28,214 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533047875_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:28,214 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533047875_0001_00_000002, Exiting
2016-05-18 00:57:28,215 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533047875_0001_00_000002
2016-05-18 00:57:28,215 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533047875_0001_00_000002
2016-05-18 00:57:28,215 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533047875_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:28,215 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533047875_0001_00_000002 completed successfully
2016-05-18 00:57:28,215 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:305A0FEF7F9E44ED96DDE1627C870214 of vertex/vertexGroup:CF42967C11B94C59AD48844062EE012E isVertexGroupOutput:false
2016-05-18 00:57:28,215 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:28,218 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533047875_0001_1, queueSize=0
2016-05-18 00:57:28,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533047875_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533047875_0001_1, startTime=1463533048044, finishTime=1463533048215, timeTaken=171, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=290, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066096307, Process_Duration=15, Process_End_Time=2927066096322, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:28,219 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533047875_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:28,219 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533047875_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:28 Completed Dag: dag_1463533047875_0001_1
2016-05-18 00:57:28,220 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533047875_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:28,220 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533047875_0001_1_post
2016-05-18 00:57:28,220 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,220 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:28,220 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:28,221 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95] every/(1/1) ...atform/-1_UNKNOWN_RESULTS, with id=dag_1463533047875_0001_1
2016-05-18 00:57:28,221 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95] every/(1/1) ...atform/-1_UNKNOWN_RESULTS, with id=dag_1463533047875_0001_1
2016-05-18 00:57:28,227 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...atform/-1_UNKNOWN_RESULTS, applicationId=application_1463533047875_0001
2016-05-18 00:57:28,229 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:28,230 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:28,230 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:28,230 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:28,230 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:28,230 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:28,231 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:28,231 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:28,231 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:28,232 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:28,232 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:28,232 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:28,232 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:28,233 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:28,237 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0B47C2D99AB1480E9857580D068035EA/E7B82B3A580D4D8C84BACDED212ADC95/.tez/application_1463533047875_0001
2016-05-18 00:57:28,238 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_RESULTS/_temporary
2016-05-18 00:57:28,238 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.390
2016-05-18 00:57:28,314 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_UNKNOWN_RESULTS
2016-05-18 00:57:28,346 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:28,347 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:28,361 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:28,361 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:28,394 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.018
2016-05-18 00:57:28,406 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.030
2016-05-18 00:57:28,407 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:28,407 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:28,407 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:28,425 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:28,432 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_ALL
2016-05-18 00:57:28,488 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:28,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:28,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_ALL"]
2016-05-18 00:57:28,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:28,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:28,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:28,489 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/-1_UNKNOWN_ALL
2016-05-18 00:57:28,490 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:28,503 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:28,507 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:28,508 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:28,509 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:28,518 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001 doesn't exist and is created
2016-05-18 00:57:28,533 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:28,533 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:28,534 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001_wd
2016-05-18 00:57:28,544 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533048508_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:28,545 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:28,545 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:28,546 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:28,547 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:28,547 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:28,547 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:28,548 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:28,549 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:28,549 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:28,550 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:28,550 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:28,550 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:28,550 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:28,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:28,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:28,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:28,551 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:28,552 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:28,552 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:28,553 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:28,554 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:28,555 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:28,555 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001_wd/localmode-log-dir
2016-05-18 00:57:28,555 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001_wd/localmode-log-dir/history.txt.appattempt_1463533048508_0001_000000, maxErrors=10
2016-05-18 00:57:28,556 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:28,556 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533048508_0001_000000, appSubmitTime=1463533048544, launchTime=1463533048544
2016-05-18 00:57:28,562 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:28,571 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:58541
2016-05-18 00:57:28,573 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533048508
2016-05-18 00:57:28,575 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533048508_0001_000000, startTime=1463533048574
2016-05-18 00:57:28,575 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:28,634 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:28,634 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:28,634 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/-1_UNKNOWN_ALL, applicationId=application_1463533048508_0001, dagName=[6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692] every/(1/1) ...lyplatform/-1_UNKNOWN_ALL
2016-05-18 00:57:28,652 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692] every/(1/1) ...lyplatform/-1_UNKNOWN_ALL
2016-05-18 00:57:28,653 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:28,654 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533048508_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001_wd/localmode-log-dir/dag_1463533048508_0001_1.dot
2016-05-18 00:57:28,654 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001_wd/localmode-log-dir/dag_1463533048508_0001_1-tez-dag.pb.txt
2016-05-18 00:57:28,668 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533048508_0001_1
2016-05-18 00:57:28,668 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,668 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692] every/(1/1) ...lyplatform/-1_UNKNOWN_ALL
2016-05-18 00:57:28 Running Dag: dag_1463533048508_0001_1
2016-05-18 00:57:28,678 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533048508_0001_1, submitTime=1463533048653
2016-05-18 00:57:28,679 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001/tez-conf.pb" } size: 36913 timestamp: 1463533048000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:28,680 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,680 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:28,686 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001/tez-conf.pb" } size: 36913 timestamp: 1463533048000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:28,686 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]
2016-05-18 00:57:28,686 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:28,686 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:28,688 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533048508_0001_1, initTime=1463533048688
2016-05-18 00:57:28,688 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533048508_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:28,688 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:28,689 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:DAG_STARTED]: dagID=dag_1463533048508_0001_1, startTime=1463533048689
2016-05-18 00:57:28,689 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533048508_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:28,689 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: EF8D2B3415D945D6A8A0F0C80B188EA8 : {CE913BC3EFD34D60AEAE7BC274051247={InputName=CE913BC3EFD34D60AEAE7BC274051247}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:28,689 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: CE913BC3EFD34D60AEAE7BC274051247, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:28,689 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,689 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,690 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,690 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,691 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:28,691 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]
2016-05-18 00:57:28,694 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/-1_UNKNOWN_ALL, applicationId=application_1463533048508_0001, dagName=[6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692] every/(1/1) ...lyplatform/-1_UNKNOWN_ALL
2016-05-18 00:57:28,695 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: CE913BC3EFD34D60AEAE7BC274051247 on vertex vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,696 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:28,697 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]
2016-05-18 00:57:28,697 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]
2016-05-18 00:57:28,698 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533048508_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:28,698 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79], numAdditionalOutputs=1
2016-05-18 00:57:28,699 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=A078A5A421154CC3AD255223B4D91442, vertex=vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:28,702 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input CE913BC3EFD34D60AEAE7BC274051247 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:28,702 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for C9831C5BC3F4412AA3A431C47A7BAD79:A078A5A421154CC3AD255223B4D91442 using oldmapred API
2016-05-18 00:57:28,703 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=A078A5A421154CC3AD255223B4D91442, vertexName=C9831C5BC3F4412AA3A431C47A7BAD79, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:28,704 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C9831C5BC3F4412AA3A431C47A7BAD79, vertexId=vertex_1463533048508_0001_1_01, initRequestedTime=1463533048691, initedTime=1463533048698, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:28,704 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:28,704 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:28,705 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:28,705 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:28,706 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:28,706 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: CE913BC3EFD34D60AEAE7BC274051247 on vertex vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,707 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:28,707 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {CE913BC3EFD34D60AEAE7BC274051247=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:28,708 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] parallelism set to 1
2016-05-18 00:57:28,708 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:28,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EF8D2B3415D945D6A8A0F0C80B188EA8, vertexId=vertex_1463533048508_0001_1_00, initRequestedTime=1463533048689, initedTime=1463533048708, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:28,709 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] with distanceFromRoot: 0
2016-05-18 00:57:28,709 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:28,710 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533048508_0001_1_00, startRequestedTime=1463533048691, startedTime=1463533048709
2016-05-18 00:57:28,710 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533048508_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:28,710 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:28,711 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in EF8D2B3415D945D6A8A0F0C80B188EA8
2016-05-18 00:57:28,711 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,711 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533048508_0001_1_00 for vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79] numStartedSources: 1 numSources: 1
2016-05-18 00:57:28,711 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79] with distanceFromRoot: 1
2016-05-18 00:57:28,712 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533048508_0001_1_01, startRequestedTime=1463533048711, startedTime=1463533048712
2016-05-18 00:57:28,713 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:28,714 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: C9831C5BC3F4412AA3A431C47A7BAD79 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:28,714 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:TASK_STARTED]: vertexName=EF8D2B3415D945D6A8A0F0C80B188EA8, taskId=task_1463533048508_0001_1_00_000000, scheduledTime=1463533048713, launchTime=1463533048713
2016-05-18 00:57:28,714 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: EF8D2B3415D945D6A8A0F0C80B188EA8 in vertex: C9831C5BC3F4412AA3A431C47A7BAD79 numBipartiteSourceTasks: 1
2016-05-18 00:57:28,715 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,716 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:28,717 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533048508_0001_00_000001, launchTime=1463533048717
2016-05-18 00:57:28,718 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533048508_0001_00_000001
2016-05-18 00:57:28,719 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533048508_0001_1_00_000000_0] started. Is using containerId: [container_1463533048508_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:28,719 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EF8D2B3415D945D6A8A0F0C80B188EA8, taskAttemptId=attempt_1463533048508_0001_1_00_000000_0, startTime=1463533048718, containerId=container_1463533048508_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:28,719 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533048508_0001, with dag id: dag_1463533048508_0001_1
2016-05-18 00:57:28,720 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533048508_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533048508_0001_1_00_000000_0
2016-05-18 00:57:28 Starting to run new task attempt: attempt_1463533048508_0001_1_00_000000_0
2016-05-18 00:57:28,722 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533048508_0001_1_00_000000_0
2016-05-18 00:57:28,722 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,722 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:28,722 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692] every/(1/1) ...lyplatform/-1_UNKNOWN_ALL, VertexName: EF8D2B3415D945D6A8A0F0C80B188EA8, VertexParallelism: 1, TaskAttemptID:attempt_1463533048508_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CE913BC3EFD34D60AEAE7BC274051247, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C9831C5BC3F4412AA3A431C47A7BAD79, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:28,723 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:28,724 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533048508_0001_1_00_000000_0
2016-05-18 00:57:28,724 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533048508_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:28,725 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533048508_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:28,728 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:28,732 INFO input.MRInput (MRInput.java:initialize(440)) - CE913BC3EFD34D60AEAE7BC274051247 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:28,733 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: CE913BC3EFD34D60AEAE7BC274051247
2016-05-18 00:57:28,736 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EF8D2B3415D945D6A8A0F0C80B188EA8, ordinal: 0
2016-05-18 00:57:28,736 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EF8D2B3415D945D6A8A0F0C80B188EA8, mem on start (mb), free: 313, total: 366, max: 672
2016-05-18 00:57:28,736 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:28,736 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:28,736 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:28,736 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:28,736 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CE913BC3EFD34D60AEAE7BC274051247:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [C9831C5BC3F4412AA3A431C47A7BAD79:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:28,737 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:28,737 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:28,737 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:28,738 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CE913BC3EFD34D60AEAE7BC274051247:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [C9831C5BC3F4412AA3A431C47A7BAD79:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:28,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:28,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CE913BC3EFD34D60AEAE7BC274051247 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:28,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CE913BC3EFD34D60AEAE7BC274051247
2016-05-18 00:57:28,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:28,739 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:28,739 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:28,739 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533048508_0001_1_00_000000_0
2016-05-18 00:57:28,742 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:28,743 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: CE913BC3EFD34D60AEAE7BC274051247
2016-05-18 00:57:28,743 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 47D6098A8D0A46CDB5E52A8C278BF88C
2016-05-18 00:57:28,744 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:28,744 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - CE913BC3EFD34D60AEAE7BC274051247 initialized RecordReader from event
2016-05-18 00:57:28,744 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 47D6098A8D0A46CDB5E52A8C278BF88C
2016-05-18 00:57:28,746 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:28,749 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:28,749 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - C9831C5BC3F4412AA3A431C47A7BAD79 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4a3a08ea, comparator=cascading.tuple.hadoop.util.TupleComparator@281f60b8, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:28,750 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for C9831C5BC3F4412AA3A431C47A7BAD79: , UsingHashComparator=false
2016-05-18 00:57:28,763 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:28,764 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:28,764 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for C9831C5BC3F4412AA3A431C47A7BAD79: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:28,764 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - C9831C5BC3F4412AA3A431C47A7BAD79: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:28,765 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:28,765 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EF8D2B3415D945D6A8A0F0C80B188EA8, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:28,768 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533048508_0001_1_00_000000_0
2016-05-18 00:57:28,768 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EF8D2B3415D945D6A8A0F0C80B188EA8, mem on close (mb), free: 211, total: 366, max: 672
2016-05-18 00:57:28,769 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - C9831C5BC3F4412AA3A431C47A7BAD79: Starting flush of map output
2016-05-18 00:57:28,769 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - C9831C5BC3F4412AA3A431C47A7BAD79: Span0.length = 20, perItem = 6
2016-05-18 00:57:28,769 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - C9831C5BC3F4412AA3A431C47A7BAD79: done sorting span=0, length=20, time=0
2016-05-18 00:57:28,770 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - C9831C5BC3F4412AA3A431C47A7BAD79: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:28,772 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - C9831C5BC3F4412AA3A431C47A7BAD79: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001_wd/localmode-local-dir/output/attempt_1463533048508_0001_1_00_000000_0_10045_0/file.out
2016-05-18 00:57:28,776 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:28 Completed running task attempt: attempt_1463533048508_0001_1_00_000000_0
2016-05-18 00:57:28,777 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533048508_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:28,777 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533048508_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:28,778 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533048508_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533048765, Process_Duration=3, Process_End_Time=1463533048768, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:28,779 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533048508_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:28,779 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:28,779 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:28,779 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,780 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EF8D2B3415D945D6A8A0F0C80B188EA8, taskAttemptId=attempt_1463533048508_0001_1_00_000000_0, creationTime=1463533048715, allocationTime=1463533048717, startTime=1463533048718, finishTime=1463533048779, timeTaken=61, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:28,780 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533048508_0001_1_00_000000_0
2016-05-18 00:57:28,781 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:TASK_FINISHED]: vertexName=EF8D2B3415D945D6A8A0F0C80B188EA8, taskId=task_1463533048508_0001_1_00_000000, startTime=1463533048718, finishTime=1463533048780, timeTaken=62, status=SUCCEEDED, successfulAttemptID=attempt_1463533048508_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533048765, Process_Duration=3, Process_End_Time=1463533048768, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:28,781 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:28,782 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,782 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533048508_0001_00_000001
2016-05-18 00:57:28,783 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:VERTEX_FINISHED]: vertexName=EF8D2B3415D945D6A8A0F0C80B188EA8, vertexId=vertex_1463533048508_0001_1_00, initRequestedTime=1463533048689, initedTime=1463533048708, startRequestedTime=1463533048691, startedTime=1463533048709, finishTime=1463533048782, timeTaken=73, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533048765, Process_Duration=3, Process_End_Time=1463533048768, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533048718, firstTasksToStart=[ task_1463533048508_0001_1_00_000000 ], lastTaskFinishTime=1463533048779, lastTasksToFinish=[ task_1463533048508_0001_1_00_000000 ], minTaskDuration=61, maxTaskDuration=61, avgTaskDuration=61.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533048508_0001_1_00_000000 ], longestDurationTasks=[ task_1463533048508_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:28,783 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:28,784 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533048508_0001_00_000001, stoppedTime=1463533048783, exitStatus=0
2016-05-18 00:57:28,784 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79] attempt: attempt_1463533048508_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:28,784 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:28,784 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C9831C5BC3F4412AA3A431C47A7BAD79
2016-05-18 00:57:28,784 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=EF8D2B3415D945D6A8A0F0C80B188EA8, destinationVertexName=C9831C5BC3F4412AA3A431C47A7BAD79. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:28,784 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]
2016-05-18 00:57:28,784 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:28,785 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533048508_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:28,786 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:TASK_STARTED]: vertexName=C9831C5BC3F4412AA3A431C47A7BAD79, taskId=task_1463533048508_0001_1_01_000000, scheduledTime=1463533048785, launchTime=1463533048785
2016-05-18 00:57:28,786 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]
2016-05-18 00:57:28,789 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533048508_0001_00_000002, launchTime=1463533048789
2016-05-18 00:57:28,790 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533048508_0001_00_000002
2016-05-18 00:57:28,790 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533048508_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533048508_0001_1_01_000000_0
2016-05-18 00:57:28,790 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533048508_0001_1_01_000000_0] started. Is using containerId: [container_1463533048508_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:28,791 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C9831C5BC3F4412AA3A431C47A7BAD79, taskAttemptId=attempt_1463533048508_0001_1_01_000000_0, startTime=1463533048790, containerId=container_1463533048508_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:28 Starting to run new task attempt: attempt_1463533048508_0001_1_01_000000_0
2016-05-18 00:57:28,791 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533048508_0001_1_01_000000_0
2016-05-18 00:57:28,792 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,792 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:28,792 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692] every/(1/1) ...lyplatform/-1_UNKNOWN_ALL, VertexName: C9831C5BC3F4412AA3A431C47A7BAD79, VertexParallelism: 1, TaskAttemptID:attempt_1463533048508_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EF8D2B3415D945D6A8A0F0C80B188EA8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A078A5A421154CC3AD255223B4D91442, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:28,793 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:28,793 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533048508_0001_1_01_000000_0
2016-05-18 00:57:28,793 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533048508_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:28,793 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533048508_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:28,794 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533048508_0001_00_000001, Exiting
2016-05-18 00:57:28,794 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533048508_0001_00_000001
2016-05-18 00:57:28,794 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533048508_0001_00_000001
2016-05-18 00:57:28,796 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533048508_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]
2016-05-18 00:57:28,797 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533048508_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:28,798 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533048508_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:28,798 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533048508_0001_00_000001 completed successfully
2016-05-18 00:57:28,801 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:28,801 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:28,810 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C9831C5BC3F4412AA3A431C47A7BAD79, ordinal: 1
2016-05-18 00:57:28,810 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C9831C5BC3F4412AA3A431C47A7BAD79, mem on start (mb), free: 206, total: 366, max: 672
2016-05-18 00:57:28,811 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:28,811 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:28,811 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:28,812 INFO output.MROutput (MROutput.java:initialize(421)) - A078A5A421154CC3AD255223B4D91442: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:28,813 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:28,813 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A078A5A421154CC3AD255223B4D91442:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [EF8D2B3415D945D6A8A0F0C80B188EA8:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:28,814 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:28,814 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:28,814 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:28,815 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A078A5A421154CC3AD255223B4D91442:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [EF8D2B3415D945D6A8A0F0C80B188EA8:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:28,815 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:28,815 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: EF8D2B3415D945D6A8A0F0C80B188EA8 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:28,816 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:28,816 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:28,816 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - EF8D2B3415D945D6A8A0F0C80B188EA8: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:28,816 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - EF8D2B3415D945D6A8A0F0C80B188EA8: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:28,818 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: EF8D2B3415D945D6A8A0F0C80B188EA8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:28,818 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: EF8D2B3415D945D6A8A0F0C80B188EA8
2016-05-18 00:57:28,819 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:28,819 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533048508_0001_1_01_000000_0
2016-05-18 00:57:28,820 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EF8D2B3415D945D6A8A0F0C80B188EA8: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:28,822 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:28,822 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533048508_0001_1_00_000000_0_10045} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533048822, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:28,823 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : EF8D2B3415D945D6A8A0F0C80B188EA8
2016-05-18 00:57:28,824 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:28,823 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 47D6098A8D0A46CDB5E52A8C278BF88C
2016-05-18 00:57:28,824 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_ALL"], id: A078A5A421154CC3AD255223B4D91442
2016-05-18 00:57:28,825 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_ALL"]
2016-05-18 00:57:28,825 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 47D6098A8D0A46CDB5E52A8C278BF88C, for 1 inputs
2016-05-18 00:57:28,824 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: EF8D2B3415D945D6A8A0F0C80B188EA8, wasInterrupted=false
2016-05-18 00:57:28,826 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:28,827 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:28,828 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:28,828 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001_wd/localmode-local-dir/output/attempt_1463533048508_0001_1_00_000000_0_10045/file.out, len=181, isLocal=true
2016-05-18 00:57:28,828 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:28,829 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:28,829 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:28,829 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:28,830 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : EF8D2B3415D945D6A8A0F0C80B188EA8
2016-05-18 00:57:28,830 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C9831C5BC3F4412AA3A431C47A7BAD79, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:28,830 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - EF8D2B3415D945D6A8A0F0C80B188EA8: Shuffle Runner thread complete
2016-05-18 00:57:28,831 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:28,831 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - EF8D2B3415D945D6A8A0F0C80B188EA8: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:28,837 INFO output.MROutput (MROutput.java:close(526)) - A078A5A421154CC3AD255223B4D91442 closed
2016-05-18 00:57:28,838 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533048508_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:28,839 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533048508_0001_1_01_000000_0
2016-05-18 00:57:28,839 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C9831C5BC3F4412AA3A431C47A7BAD79, mem on close (mb), free: 202, total: 366, max: 672
2016-05-18 00:57:28,839 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: EF8D2B3415D945D6A8A0F0C80B188EA8
2016-05-18 00:57:28,839 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EF8D2B3415D945D6A8A0F0C80B188EA8: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:28,839 INFO output.MROutput (MROutput.java:close(526)) - A078A5A421154CC3AD255223B4D91442 closed
2016-05-18 00:57:28,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:28 Completed running task attempt: attempt_1463533048508_0001_1_01_000000_0
2016-05-18 00:57:28,840 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533048508_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:28,840 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533048508_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:28,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533048508_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533048830, Process_Duration=9, Process_End_Time=1463533048839, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:28,841 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533048508_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:28,841 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:28,842 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:28,842 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C9831C5BC3F4412AA3A431C47A7BAD79, taskAttemptId=attempt_1463533048508_0001_1_01_000000_0, creationTime=1463533048786, allocationTime=1463533048788, startTime=1463533048790, finishTime=1463533048842, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:28,842 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533048508_0001_1_01_000000_0
2016-05-18 00:57:28,842 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,843 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533048508_0001_00_000002
2016-05-18 00:57:28,843 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:TASK_FINISHED]: vertexName=C9831C5BC3F4412AA3A431C47A7BAD79, taskId=task_1463533048508_0001_1_01_000000, startTime=1463533048790, finishTime=1463533048842, timeTaken=52, status=SUCCEEDED, successfulAttemptID=attempt_1463533048508_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533048830, Process_Duration=9, Process_End_Time=1463533048839, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:28,844 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:28,844 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79]
2016-05-18 00:57:28,845 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:VERTEX_FINISHED]: vertexName=C9831C5BC3F4412AA3A431C47A7BAD79, vertexId=vertex_1463533048508_0001_1_01, initRequestedTime=1463533048691, initedTime=1463533048698, startRequestedTime=1463533048711, startedTime=1463533048712, finishTime=1463533048844, timeTaken=132, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533048830, Process_Duration=9, Process_End_Time=1463533048839, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533048790, firstTasksToStart=[ task_1463533048508_0001_1_01_000000 ], lastTaskFinishTime=1463533048842, lastTasksToFinish=[ task_1463533048508_0001_1_01_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533048508_0001_1_01_000000 ], longestDurationTasks=[ task_1463533048508_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:28,845 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:28,846 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533048508_0001_00_000002, stoppedTime=1463533048846, exitStatus=0
2016-05-18 00:57:28,846 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:28,846 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:28,846 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533048508_0001_1_00 [EF8D2B3415D945D6A8A0F0C80B188EA8]
2016-05-18 00:57:28,846 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533048508_0001_1
2016-05-18 00:57:28,847 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533048508_0001_1
2016-05-18 00:57:28,847 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: A078A5A421154CC3AD255223B4D91442 for vertex: vertex_1463533048508_0001_1_01 [C9831C5BC3F4412AA3A431C47A7BAD79], outputName: A078A5A421154CC3AD255223B4D91442
2016-05-18 00:57:28,847 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533048508_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:28,847 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533048508_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:28,854 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533048508_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:28,854 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:A078A5A421154CC3AD255223B4D91442 of vertex/vertexGroup:C9831C5BC3F4412AA3A431C47A7BAD79 isVertexGroupOutput:false
2016-05-18 00:57:28,854 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:28,854 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533048508_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:28,855 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533048508_0001_00_000002, Exiting
2016-05-18 00:57:28,855 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533048508_0001_00_000002
2016-05-18 00:57:28,855 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533048508_0001_00_000002
2016-05-18 00:57:28,855 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533048508_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:28,855 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533048508_0001_00_000002 completed successfully
2016-05-18 00:57:28,857 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533048508_0001_1, queueSize=0
2016-05-18 00:57:28,858 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533048508_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533048508_0001_1, startTime=1463533048688, finishTime=1463533048854, timeTaken=166, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066097595, Process_Duration=12, Process_End_Time=2927066097607, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:28,858 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533048508_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:28,858 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533048508_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:28 Completed Dag: dag_1463533048508_0001_1
2016-05-18 00:57:28,858 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533048508_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:28,858 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533048508_0001_1_post
2016-05-18 00:57:28,859 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:28,859 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:28,859 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:28,859 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692] every/(1/1) ...lyplatform/-1_UNKNOWN_ALL, with id=dag_1463533048508_0001_1
2016-05-18 00:57:28,859 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692] every/(1/1) ...lyplatform/-1_UNKNOWN_ALL, with id=dag_1463533048508_0001_1
2016-05-18 00:57:28,867 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/-1_UNKNOWN_ALL, applicationId=application_1463533048508_0001
2016-05-18 00:57:28,869 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:28,869 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:28,869 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:28,869 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:28,869 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:28,870 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:28,870 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:28,871 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:28,871 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:28,871 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:28,872 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:28,872 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:28,872 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:28,873 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:28,876 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6CCAC807107C43A8921D746F16B8F536/BB7DB611602D49EB993A31D9F07D2692/.tez/application_1463533048508_0001
2016-05-18 00:57:28,876 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_ALL/_temporary
2016-05-18 00:57:28,877 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.388
2016-05-18 00:57:28,952 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_UNKNOWN_ALL
2016-05-18 00:57:28,975 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:28,975 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:29,015 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:29,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.028
2016-05-18 00:57:29,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:29,031 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:29,031 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:29,047 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:29,054 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_REPLACE
2016-05-18 00:57:29,106 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:29,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:29,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_REPLACE"]
2016-05-18 00:57:29,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:29,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:29,108 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:29,110 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...atform/-1_UNKNOWN_REPLACE
2016-05-18 00:57:29,110 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:29,126 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:29,130 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:29,131 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:29,131 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:29,142 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001 doesn't exist and is created
2016-05-18 00:57:29,156 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:29,157 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:29,158 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001_wd
2016-05-18 00:57:29,170 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533049131_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:29,170 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:29,171 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:29,173 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:29,173 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:29,173 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:29,173 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:29,174 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:29,175 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:29,175 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:29,175 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:29,175 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:29,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:29,176 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:29,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:29,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:29,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:29,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:29,177 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:29,177 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:29,178 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:29,179 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:29,179 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:29,179 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001_wd/localmode-log-dir
2016-05-18 00:57:29,179 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001_wd/localmode-log-dir/history.txt.appattempt_1463533049131_0001_000000, maxErrors=10
2016-05-18 00:57:29,180 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:29,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533049131_0001_000000, appSubmitTime=1463533049170, launchTime=1463533049170
2016-05-18 00:57:29,192 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:29,196 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:39797
2016-05-18 00:57:29,198 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533049131
2016-05-18 00:57:29,200 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533049131_0001_000000, startTime=1463533049200
2016-05-18 00:57:29,200 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:29,257 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:29,257 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:29,257 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...atform/-1_UNKNOWN_REPLACE, applicationId=application_1463533049131_0001, dagName=[D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706] every/(1/1) ...atform/-1_UNKNOWN_REPLACE
2016-05-18 00:57:29,275 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706] every/(1/1) ...atform/-1_UNKNOWN_REPLACE
2016-05-18 00:57:29,276 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:29,276 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533049131_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001_wd/localmode-log-dir/dag_1463533049131_0001_1.dot
2016-05-18 00:57:29,277 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001_wd/localmode-log-dir/dag_1463533049131_0001_1-tez-dag.pb.txt
2016-05-18 00:57:29,283 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533049131_0001_1
2016-05-18 00:57:29,284 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:29,284 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706] every/(1/1) ...atform/-1_UNKNOWN_REPLACE
2016-05-18 00:57:29 Running Dag: dag_1463533049131_0001_1
2016-05-18 00:57:29,294 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533049131_0001_1, submitTime=1463533049275
2016-05-18 00:57:29,295 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001/tez-conf.pb" } size: 36913 timestamp: 1463533049000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:29,295 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,295 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:29,296 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001/tez-conf.pb" } size: 36913 timestamp: 1463533049000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:29,296 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,296 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:29,296 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:29,297 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533049131_0001_1, initTime=1463533049296
2016-05-18 00:57:29,297 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533049131_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:29,297 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:29,298 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:DAG_STARTED]: dagID=dag_1463533049131_0001_1, startTime=1463533049297
2016-05-18 00:57:29,298 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533049131_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:29,298 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F2AD328E3DDE40F4B245FB5015437F55 : {3080C0F5285C4449AF1CEF25675D36F5={InputName=3080C0F5285C4449AF1CEF25675D36F5}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:29,298 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 3080C0F5285C4449AF1CEF25675D36F5, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:29,298 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,298 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,299 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,299 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...atform/-1_UNKNOWN_REPLACE, applicationId=application_1463533049131_0001, dagName=[D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706] every/(1/1) ...atform/-1_UNKNOWN_REPLACE
2016-05-18 00:57:29,299 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,299 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:29,300 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,299 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 3080C0F5285C4449AF1CEF25675D36F5 on vertex vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,302 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 3080C0F5285C4449AF1CEF25675D36F5 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:29,303 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:29,303 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,304 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,304 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533049131_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:29,305 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4], numAdditionalOutputs=1
2016-05-18 00:57:29,305 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=ED0CE3DAEBFA485EA7C36DA4C9E498BC, vertex=vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:29,305 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:29,306 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:29,306 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:29,306 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:29,307 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 3080C0F5285C4449AF1CEF25675D36F5 on vertex vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,307 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DCC72D2EACB0471DB5D73683FE6BB7C4:ED0CE3DAEBFA485EA7C36DA4C9E498BC using oldmapred API
2016-05-18 00:57:29,308 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=ED0CE3DAEBFA485EA7C36DA4C9E498BC, vertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:29,309 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, vertexId=vertex_1463533049131_0001_1_01, initRequestedTime=1463533049300, initedTime=1463533049305, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:29,309 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:29,311 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:29,311 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {3080C0F5285C4449AF1CEF25675D36F5=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:29,311 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] parallelism set to 1
2016-05-18 00:57:29,312 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:29,313 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F2AD328E3DDE40F4B245FB5015437F55, vertexId=vertex_1463533049131_0001_1_00, initRequestedTime=1463533049298, initedTime=1463533049313, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:29,313 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] with distanceFromRoot: 0
2016-05-18 00:57:29,314 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:29,314 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533049131_0001_1_00, startRequestedTime=1463533049300, startedTime=1463533049314
2016-05-18 00:57:29,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533049131_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:29,315 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:29,315 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533049131_0001_1_00 for vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] numStartedSources: 1 numSources: 1
2016-05-18 00:57:29,315 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] with distanceFromRoot: 1
2016-05-18 00:57:29,315 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F2AD328E3DDE40F4B245FB5015437F55
2016-05-18 00:57:29,316 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,317 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533049131_0001_1_01, startRequestedTime=1463533049315, startedTime=1463533049316
2016-05-18 00:57:29,319 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:29,321 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:TASK_STARTED]: vertexName=F2AD328E3DDE40F4B245FB5015437F55, taskId=task_1463533049131_0001_1_00_000000, scheduledTime=1463533049320, launchTime=1463533049320
2016-05-18 00:57:29,321 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DCC72D2EACB0471DB5D73683FE6BB7C4 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:29,321 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,322 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F2AD328E3DDE40F4B245FB5015437F55 in vertex: DCC72D2EACB0471DB5D73683FE6BB7C4 numBipartiteSourceTasks: 1
2016-05-18 00:57:29,322 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:29,324 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533049131_0001_00_000001, launchTime=1463533049323
2016-05-18 00:57:29,324 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533049131_0001_00_000001
2016-05-18 00:57:29,325 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533049131_0001_1_00_000000_0] started. Is using containerId: [container_1463533049131_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:29,326 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F2AD328E3DDE40F4B245FB5015437F55, taskAttemptId=attempt_1463533049131_0001_1_00_000000_0, startTime=1463533049325, containerId=container_1463533049131_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:29,326 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533049131_0001, with dag id: dag_1463533049131_0001_1
2016-05-18 00:57:29,328 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533049131_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533049131_0001_1_00_000000_0
2016-05-18 00:57:29 Starting to run new task attempt: attempt_1463533049131_0001_1_00_000000_0
2016-05-18 00:57:29,330 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533049131_0001_1_00_000000_0
2016-05-18 00:57:29,330 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:29,330 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:29,330 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706] every/(1/1) ...atform/-1_UNKNOWN_REPLACE, VertexName: F2AD328E3DDE40F4B245FB5015437F55, VertexParallelism: 1, TaskAttemptID:attempt_1463533049131_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=3080C0F5285C4449AF1CEF25675D36F5, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:29,331 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:29,331 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533049131_0001_1_00_000000_0
2016-05-18 00:57:29,332 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533049131_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:29,332 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533049131_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:29,336 INFO input.MRInput (MRInput.java:initialize(440)) - 3080C0F5285C4449AF1CEF25675D36F5 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:29,337 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 3080C0F5285C4449AF1CEF25675D36F5
2016-05-18 00:57:29,337 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:29,343 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F2AD328E3DDE40F4B245FB5015437F55, ordinal: 0
2016-05-18 00:57:29,344 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F2AD328E3DDE40F4B245FB5015437F55, mem on start (mb), free: 310, total: 366, max: 672
2016-05-18 00:57:29,344 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:29,344 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:29,344 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:29,344 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:29,344 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[3080C0F5285C4449AF1CEF25675D36F5:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [DCC72D2EACB0471DB5D73683FE6BB7C4:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:29,344 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:29,344 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:29,345 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:29,345 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[3080C0F5285C4449AF1CEF25675D36F5:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [DCC72D2EACB0471DB5D73683FE6BB7C4:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:29,345 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:29,345 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 3080C0F5285C4449AF1CEF25675D36F5 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:29,345 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 3080C0F5285C4449AF1CEF25675D36F5
2016-05-18 00:57:29,345 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:29,345 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:29,345 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:29,346 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533049131_0001_1_00_000000_0
2016-05-18 00:57:29,348 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:29,349 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 3080C0F5285C4449AF1CEF25675D36F5
2016-05-18 00:57:29,349 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:29,349 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 3080C0F5285C4449AF1CEF25675D36F5 initialized RecordReader from event
2016-05-18 00:57:29,349 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 2003EDFBB6F142CE96B16FAE450CC4EA
2016-05-18 00:57:29,350 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 2003EDFBB6F142CE96B16FAE450CC4EA
2016-05-18 00:57:29,352 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:29,356 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:29,356 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DCC72D2EACB0471DB5D73683FE6BB7C4 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4366712a, comparator=cascading.tuple.hadoop.util.TupleComparator@7a8f9ef7, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:29,356 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DCC72D2EACB0471DB5D73683FE6BB7C4: , UsingHashComparator=false
2016-05-18 00:57:29,367 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:29,367 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:29,369 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:29,369 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:29,369 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DCC72D2EACB0471DB5D73683FE6BB7C4: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:29,369 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DCC72D2EACB0471DB5D73683FE6BB7C4: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:29,369 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:29,369 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F2AD328E3DDE40F4B245FB5015437F55, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:29,372 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533049131_0001_1_00_000000_0
2016-05-18 00:57:29,372 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F2AD328E3DDE40F4B245FB5015437F55, mem on close (mb), free: 208, total: 366, max: 672
2016-05-18 00:57:29,372 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DCC72D2EACB0471DB5D73683FE6BB7C4: Starting flush of map output
2016-05-18 00:57:29,372 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DCC72D2EACB0471DB5D73683FE6BB7C4: Span0.length = 20, perItem = 6
2016-05-18 00:57:29,373 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DCC72D2EACB0471DB5D73683FE6BB7C4: done sorting span=0, length=20, time=1
2016-05-18 00:57:29,373 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DCC72D2EACB0471DB5D73683FE6BB7C4: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:29,374 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DCC72D2EACB0471DB5D73683FE6BB7C4: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001_wd/localmode-local-dir/output/attempt_1463533049131_0001_1_00_000000_0_10051_0/file.out
2016-05-18 00:57:29,378 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:29 Completed running task attempt: attempt_1463533049131_0001_1_00_000000_0
2016-05-18 00:57:29,379 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533049131_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:29,379 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533049131_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:29,380 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533049131_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533049370, Process_Duration=2, Process_End_Time=1463533049372, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:29,381 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533049131_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:29,381 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:29,381 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F2AD328E3DDE40F4B245FB5015437F55, taskAttemptId=attempt_1463533049131_0001_1_00_000000_0, creationTime=1463533049321, allocationTime=1463533049323, startTime=1463533049325, finishTime=1463533049381, timeTaken=56, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:29,381 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:29,382 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:29,382 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533049131_0001_1_00_000000_0
2016-05-18 00:57:29,383 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533049131_0001_00_000001
2016-05-18 00:57:29,383 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:TASK_FINISHED]: vertexName=F2AD328E3DDE40F4B245FB5015437F55, taskId=task_1463533049131_0001_1_00_000000, startTime=1463533049325, finishTime=1463533049383, timeTaken=58, status=SUCCEEDED, successfulAttemptID=attempt_1463533049131_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533049370, Process_Duration=2, Process_End_Time=1463533049372, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:29,383 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:29,383 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55]
2016-05-18 00:57:29,385 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:VERTEX_FINISHED]: vertexName=F2AD328E3DDE40F4B245FB5015437F55, vertexId=vertex_1463533049131_0001_1_00, initRequestedTime=1463533049298, initedTime=1463533049313, startRequestedTime=1463533049300, startedTime=1463533049314, finishTime=1463533049384, timeTaken=70, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533049370, Process_Duration=2, Process_End_Time=1463533049372, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533049325, firstTasksToStart=[ task_1463533049131_0001_1_00_000000 ], lastTaskFinishTime=1463533049381, lastTasksToFinish=[ task_1463533049131_0001_1_00_000000 ], minTaskDuration=56, maxTaskDuration=56, avgTaskDuration=56.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533049131_0001_1_00_000000 ], longestDurationTasks=[ task_1463533049131_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:29,385 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:29,385 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533049131_0001_00_000001, stoppedTime=1463533049385, exitStatus=0
2016-05-18 00:57:29,385 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] attempt: attempt_1463533049131_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:29,386 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DCC72D2EACB0471DB5D73683FE6BB7C4
2016-05-18 00:57:29,386 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F2AD328E3DDE40F4B245FB5015437F55, destinationVertexName=DCC72D2EACB0471DB5D73683FE6BB7C4. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:29,386 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,387 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533049131_0001_1_00 [F2AD328E3DDE40F4B245FB5015437F55] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:29,387 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:29,387 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:TASK_STARTED]: vertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, taskId=task_1463533049131_0001_1_01_000000, scheduledTime=1463533049387, launchTime=1463533049387
2016-05-18 00:57:29,387 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533049131_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:29,388 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,389 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533049131_0001_00_000002, launchTime=1463533049389
2016-05-18 00:57:29,390 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533049131_0001_00_000002
2016-05-18 00:57:29,391 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533049131_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533049131_0001_1_01_000000_0
2016-05-18 00:57:29,391 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533049131_0001_1_01_000000_0] started. Is using containerId: [container_1463533049131_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:29 Starting to run new task attempt: attempt_1463533049131_0001_1_01_000000_0
2016-05-18 00:57:29,392 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533049131_0001_1_01_000000_0
2016-05-18 00:57:29,392 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:29,392 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, taskAttemptId=attempt_1463533049131_0001_1_01_000000_0, startTime=1463533049391, containerId=container_1463533049131_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:29,392 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:29,393 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533049131_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:29,393 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533049131_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:29,393 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533049131_0001_00_000001, Exiting
2016-05-18 00:57:29,393 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533049131_0001_00_000001
2016-05-18 00:57:29,393 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533049131_0001_00_000001
2016-05-18 00:57:29,394 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706] every/(1/1) ...atform/-1_UNKNOWN_REPLACE, VertexName: DCC72D2EACB0471DB5D73683FE6BB7C4, VertexParallelism: 1, TaskAttemptID:attempt_1463533049131_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F2AD328E3DDE40F4B245FB5015437F55, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=ED0CE3DAEBFA485EA7C36DA4C9E498BC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:29,394 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:29,395 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533049131_0001_1_01_000000_0
2016-05-18 00:57:29,395 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533049131_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,395 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533049131_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:29,395 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533049131_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:29,396 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533049131_0001_00_000001 completed successfully
2016-05-18 00:57:29,398 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:29,400 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:29,407 INFO output.MROutput (MROutput.java:initialize(421)) - ED0CE3DAEBFA485EA7C36DA4C9E498BC: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:29,408 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DCC72D2EACB0471DB5D73683FE6BB7C4, ordinal: 1
2016-05-18 00:57:29,408 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DCC72D2EACB0471DB5D73683FE6BB7C4, mem on start (mb), free: 202, total: 366, max: 672
2016-05-18 00:57:29,408 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:29,408 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:29,409 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:29,409 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:29,409 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F2AD328E3DDE40F4B245FB5015437F55:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [ED0CE3DAEBFA485EA7C36DA4C9E498BC:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:29,409 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:29,409 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:29,410 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:29,410 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F2AD328E3DDE40F4B245FB5015437F55:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [ED0CE3DAEBFA485EA7C36DA4C9E498BC:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:29,410 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:29,410 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F2AD328E3DDE40F4B245FB5015437F55 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:29,413 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:29,413 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:29,413 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F2AD328E3DDE40F4B245FB5015437F55: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:29,414 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F2AD328E3DDE40F4B245FB5015437F55: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:29,415 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F2AD328E3DDE40F4B245FB5015437F55 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:29,417 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F2AD328E3DDE40F4B245FB5015437F55
2016-05-18 00:57:29,417 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:29,418 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533049131_0001_1_01_000000_0
2016-05-18 00:57:29,419 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F2AD328E3DDE40F4B245FB5015437F55: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:29,420 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533049131_0001_1_00_000000_0_10051} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533049420, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:29,420 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F2AD328E3DDE40F4B245FB5015437F55
2016-05-18 00:57:29,421 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:29,421 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:29,421 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F2AD328E3DDE40F4B245FB5015437F55, wasInterrupted=false
2016-05-18 00:57:29,422 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 2003EDFBB6F142CE96B16FAE450CC4EA
2016-05-18 00:57:29,422 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:29,422 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_REPLACE"], id: ED0CE3DAEBFA485EA7C36DA4C9E498BC
2016-05-18 00:57:29,422 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - F2AD328E3DDE40F4B245FB5015437F55: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:29,422 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:29,422 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_REPLACE"]
2016-05-18 00:57:29,423 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 2003EDFBB6F142CE96B16FAE450CC4EA, for 1 inputs
2016-05-18 00:57:29,424 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:29,424 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001_wd/localmode-local-dir/output/attempt_1463533049131_0001_1_00_000000_0_10051/file.out, len=181, isLocal=true
2016-05-18 00:57:29,424 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:29,424 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:29,424 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:29,424 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:29,424 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F2AD328E3DDE40F4B245FB5015437F55
2016-05-18 00:57:29,424 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F2AD328E3DDE40F4B245FB5015437F55: Shuffle Runner thread complete
2016-05-18 00:57:29,425 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DCC72D2EACB0471DB5D73683FE6BB7C4, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:57:29,426 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:29,427 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F2AD328E3DDE40F4B245FB5015437F55: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:29,429 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:29,434 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:29,439 INFO output.MROutput (MROutput.java:close(526)) - ED0CE3DAEBFA485EA7C36DA4C9E498BC closed
2016-05-18 00:57:29,440 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533049131_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:29,441 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533049131_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:29,441 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533049131_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533049425, Process_Duration=16, Process_End_Time=1463533049441, Read_Duration=1, Tuples_Read=1]]
2016-05-18 00:57:29,441 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:29,442 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DCC72D2EACB0471DB5D73683FE6BB7C4, mem on close (mb), free: 198, total: 366, max: 672
2016-05-18 00:57:29,442 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=DCC72D2EACB0471DB5D73683FE6BB7C4, index=1, interruptedStatus=false
2016-05-18 00:57:29,443 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F2AD328E3DDE40F4B245FB5015437F55
2016-05-18 00:57:29,443 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F2AD328E3DDE40F4B245FB5015437F55: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:29,443 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=DCC72D2EACB0471DB5D73683FE6BB7C4, sourceVertex=F2AD328E3DDE40F4B245FB5015437F55, interruptedStatus=true
2016-05-18 00:57:29,444 INFO output.MROutput (MROutput.java:close(526)) - ED0CE3DAEBFA485EA7C36DA4C9E498BC closed
2016-05-18 00:57:29,444 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=DCC72D2EACB0471DB5D73683FE6BB7C4, sourceVertex=ED0CE3DAEBFA485EA7C36DA4C9E498BC, interruptedStatus=true
2016-05-18 00:57:29,448 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533049131_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], containerShutdownRequested=false}
2016-05-18 00:57:29,449 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], containerShutdownRequested=false}
2016-05-18 00:57:29,449 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533049131_0001_00_000002
2016-05-18 00:57:29,450 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533049131_0001_00_000002
2016-05-18 00:57:29,450 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533049131_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]'}
2016-05-18 00:57:29,450 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533049131_0001_00_000002 completed but with errors
2016-05-18 00:57:29,451 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, taskAttemptId=attempt_1463533049131_0001_1_01_000000_0, creationTime=1463533049388, allocationTime=1463533049389, startTime=1463533049391, finishTime=1463533049448, timeTaken=57, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533049131_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533049425, Process_Duration=16, Process_End_Time=1463533049441, Read_Duration=1, Tuples_Read=1
2016-05-18 00:57:29,455 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533049131_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
2016-05-18 00:57:29,455 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533049131_0001_00_000002, stoppedTime=1463533049455, exitStatus=1
2016-05-18 00:57:29,455 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533049131_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:29,456 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:TASK_FINISHED]: vertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, taskId=task_1463533049131_0001_1_01_000000, startTime=1463533049391, finishTime=1463533049456, timeTaken=65, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533049131_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:29,465 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] because task failed: task_1463533049131_0001_1_01_000000
2016-05-18 00:57:29,466 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:29,466 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:29,467 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:29,467 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,468 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=ED0CE3DAEBFA485EA7C36DA4C9E498BC, vertexId=vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,472 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:VERTEX_FINISHED]: vertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, vertexId=vertex_1463533049131_0001_1_01, initRequestedTime=1463533049300, initedTime=1463533049305, startRequestedTime=1463533049315, startedTime=1463533049316, finishTime=1463533049472, timeTaken=156, status=FAILED, diagnostics=Task failed, taskId=task_1463533049131_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533049131_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533049391, firstTasksToStart=[ task_1463533049131_0001_1_01_000000 ], lastTaskFinishTime=1463533049448, lastTasksToFinish=[ task_1463533049131_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:29,474 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:29,474 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533049131_0001_1_01_000000_0 failed: true container: container_1463533049131_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:29,477 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:29,477 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:29,477 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:29,479 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533049131_0001_1, queueSize=0
2016-05-18 00:57:29,479 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049131_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533049131_0001_1, startTime=1463533049297, finishTime=1463533049478, timeTaken=181, status=FAILED, diagnostics=Vertex failed, vertexName=DCC72D2EACB0471DB5D73683FE6BB7C4, vertexId=vertex_1463533049131_0001_1_01, diagnostics=[Task failed, taskId=task_1463533049131_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533049131_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=330, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533049370, Process_Duration=2, Process_End_Time=1463533049372, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:29,482 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533049131_0001_1_01 [DCC72D2EACB0471DB5D73683FE6BB7C4]
2016-05-18 00:57:29,483 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533049131_0001_1 finished with state: FAILED
2016-05-18 00:57:29,483 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533049131_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:29 Completed Dag: dag_1463533049131_0001_1
2016-05-18 00:57:29,487 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533049131_0001_1, dagState=FAILED
2016-05-18 00:57:29,487 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533049131_0001_1_post
2016-05-18 00:57:29,487 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:29,488 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:29,488 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:29,489 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706] every/(1/1) ...atform/-1_UNKNOWN_REPLACE, with id=dag_1463533049131_0001_1
2016-05-18 00:57:29,489 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...atform/-1_UNKNOWN_REPLACE, applicationId=application_1463533049131_0001
2016-05-18 00:57:29,490 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706] every/(1/1) ...atform/-1_UNKNOWN_REPLACE, with id=dag_1463533049131_0001_1
2016-05-18 00:57:29,492 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:29,493 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:29,493 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:29,493 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:29,493 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:29,495 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:29,495 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:29,497 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:29,497 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:29,498 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:29,498 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:29,498 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:29,498 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:29,499 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:29,503 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D9167C0FB3ED471E8595F8A60B49E255/B44340DE8F874B67A45F32D0B79D8706/.tez/application_1463533049131_0001
2016-05-18 00:57:29,504 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:29,504 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...atform/-1_UNKNOWN_REPLACE
2016-05-18 00:57:29,506 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:29,506 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_REPLACE/_temporary
2016-05-18 00:57:29,506 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.400
2016-05-18 00:57:29,507 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_UNKNOWN_REPLACE
2016-05-18 00:57:29,532 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:29,532 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:29,577 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.018
2016-05-18 00:57:29,589 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.030
2016-05-18 00:57:29,589 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:29,590 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:29,590 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:29,615 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:29,625 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_SWAP
2016-05-18 00:57:29,703 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:29,704 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:29,704 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_SWAP"]
2016-05-18 00:57:29,706 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:29,706 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:29,707 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:29,709 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...yplatform/-1_UNKNOWN_SWAP
2016-05-18 00:57:29,709 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:29,729 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:29,735 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:29,736 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:29,736 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:29,748 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001 doesn't exist and is created
2016-05-18 00:57:29,764 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:29,765 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:29,766 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001_wd
2016-05-18 00:57:29,777 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533049735_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:29,778 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:29,778 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:29,779 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:29,779 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:29,779 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:29,780 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:29,780 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:29,782 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:29,782 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:29,782 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:29,782 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:29,782 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:29,782 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:29,782 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:29,782 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:29,782 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:29,783 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:29,783 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:29,783 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:29,784 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:29,785 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:29,786 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:29,786 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001_wd/localmode-log-dir
2016-05-18 00:57:29,786 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001_wd/localmode-log-dir/history.txt.appattempt_1463533049735_0001_000000, maxErrors=10
2016-05-18 00:57:29,786 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:29,787 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533049735_0001_000000, appSubmitTime=1463533049777, launchTime=1463533049777
2016-05-18 00:57:29,792 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:29,797 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:52845
2016-05-18 00:57:29,798 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533049735
2016-05-18 00:57:29,800 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533049735_0001_000000, startTime=1463533049800
2016-05-18 00:57:29,800 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:29,865 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:29,865 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:29,865 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...yplatform/-1_UNKNOWN_SWAP, applicationId=application_1463533049735_0001, dagName=[896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F] every/(1/1) ...yplatform/-1_UNKNOWN_SWAP
2016-05-18 00:57:29,885 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F] every/(1/1) ...yplatform/-1_UNKNOWN_SWAP
2016-05-18 00:57:29,886 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:29,886 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533049735_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001_wd/localmode-log-dir/dag_1463533049735_0001_1.dot
2016-05-18 00:57:29,887 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001_wd/localmode-log-dir/dag_1463533049735_0001_1-tez-dag.pb.txt
2016-05-18 00:57:29,893 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533049735_0001_1
2016-05-18 00:57:29,893 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:29,893 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F] every/(1/1) ...yplatform/-1_UNKNOWN_SWAP
2016-05-18 00:57:29 Running Dag: dag_1463533049735_0001_1
2016-05-18 00:57:29,903 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533049735_0001_1, submitTime=1463533049886
2016-05-18 00:57:29,904 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001/tez-conf.pb" } size: 36913 timestamp: 1463533049000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:29,904 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,905 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:29,906 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001/tez-conf.pb" } size: 36913 timestamp: 1463533049000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:29,914 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]
2016-05-18 00:57:29,914 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:29,914 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:29,915 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533049735_0001_1, initTime=1463533049915
2016-05-18 00:57:29,915 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533049735_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:29,916 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:29,916 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:DAG_STARTED]: dagID=dag_1463533049735_0001_1, startTime=1463533049916
2016-05-18 00:57:29,916 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533049735_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:29,916 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: DB98F868A5644A0AB164B88DBF74EBFB : {CE9AE8AAB6D343ACA24F0F41283F533A={InputName=CE9AE8AAB6D343ACA24F0F41283F533A}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:29,916 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: CE9AE8AAB6D343ACA24F0F41283F533A, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:29,916 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,916 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,917 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,917 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,918 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:29,918 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]
2016-05-18 00:57:29,919 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...yplatform/-1_UNKNOWN_SWAP, applicationId=application_1463533049735_0001, dagName=[896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F] every/(1/1) ...yplatform/-1_UNKNOWN_SWAP
2016-05-18 00:57:29,922 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: CE9AE8AAB6D343ACA24F0F41283F533A on vertex vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,922 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:29,923 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]
2016-05-18 00:57:29,923 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]
2016-05-18 00:57:29,924 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533049735_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:29,925 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440], numAdditionalOutputs=1
2016-05-18 00:57:29,925 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=8EAD7471BB924E35B52870B86F8A1599, vertex=vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:29,927 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input CE9AE8AAB6D343ACA24F0F41283F533A asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:29,928 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for BCB8ACC9695A43EEA14958B37DE41440:8EAD7471BB924E35B52870B86F8A1599 using oldmapred API
2016-05-18 00:57:29,929 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=8EAD7471BB924E35B52870B86F8A1599, vertexName=BCB8ACC9695A43EEA14958B37DE41440, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:29,930 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BCB8ACC9695A43EEA14958B37DE41440, vertexId=vertex_1463533049735_0001_1_01, initRequestedTime=1463533049918, initedTime=1463533049925, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:29,931 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:29,931 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:29,931 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:29,933 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:29,933 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:29,933 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: CE9AE8AAB6D343ACA24F0F41283F533A on vertex vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,935 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:29,936 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {CE9AE8AAB6D343ACA24F0F41283F533A=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:29,936 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] parallelism set to 1
2016-05-18 00:57:29,937 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:29,937 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DB98F868A5644A0AB164B88DBF74EBFB, vertexId=vertex_1463533049735_0001_1_00, initRequestedTime=1463533049916, initedTime=1463533049937, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:29,938 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] with distanceFromRoot: 0
2016-05-18 00:57:29,938 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:29,938 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533049735_0001_1_00, startRequestedTime=1463533049918, startedTime=1463533049938
2016-05-18 00:57:29,938 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533049735_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:29,939 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:29,939 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533049735_0001_1_00 for vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440] numStartedSources: 1 numSources: 1
2016-05-18 00:57:29,939 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440] with distanceFromRoot: 1
2016-05-18 00:57:29,939 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in DB98F868A5644A0AB164B88DBF74EBFB
2016-05-18 00:57:29,939 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,941 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533049735_0001_1_01, startRequestedTime=1463533049939, startedTime=1463533049939
2016-05-18 00:57:29,942 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:29,942 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:TASK_STARTED]: vertexName=DB98F868A5644A0AB164B88DBF74EBFB, taskId=task_1463533049735_0001_1_00_000000, scheduledTime=1463533049942, launchTime=1463533049942
2016-05-18 00:57:29,943 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:29,946 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: BCB8ACC9695A43EEA14958B37DE41440 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:29,948 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: DB98F868A5644A0AB164B88DBF74EBFB in vertex: BCB8ACC9695A43EEA14958B37DE41440 numBipartiteSourceTasks: 1
2016-05-18 00:57:29,948 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:29,951 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533049735_0001_00_000001, launchTime=1463533049951
2016-05-18 00:57:29,951 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533049735_0001_00_000001
2016-05-18 00:57:29,952 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533049735_0001, with dag id: dag_1463533049735_0001_1
2016-05-18 00:57:29,953 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533049735_0001_1_00_000000_0] started. Is using containerId: [container_1463533049735_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:29,953 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DB98F868A5644A0AB164B88DBF74EBFB, taskAttemptId=attempt_1463533049735_0001_1_00_000000_0, startTime=1463533049952, containerId=container_1463533049735_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:29,957 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533049735_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533049735_0001_1_00_000000_0
2016-05-18 00:57:29 Starting to run new task attempt: attempt_1463533049735_0001_1_00_000000_0
2016-05-18 00:57:29,958 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533049735_0001_1_00_000000_0
2016-05-18 00:57:29,958 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:29,958 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:29,960 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F] every/(1/1) ...yplatform/-1_UNKNOWN_SWAP, VertexName: DB98F868A5644A0AB164B88DBF74EBFB, VertexParallelism: 1, TaskAttemptID:attempt_1463533049735_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CE9AE8AAB6D343ACA24F0F41283F533A, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=BCB8ACC9695A43EEA14958B37DE41440, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:29,960 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:29,961 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533049735_0001_1_00_000000_0
2016-05-18 00:57:29,961 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533049735_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:29,962 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533049735_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:29,965 INFO input.MRInput (MRInput.java:initialize(440)) - CE9AE8AAB6D343ACA24F0F41283F533A using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:29,966 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: CE9AE8AAB6D343ACA24F0F41283F533A
2016-05-18 00:57:29,967 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:29,972 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DB98F868A5644A0AB164B88DBF74EBFB, ordinal: 0
2016-05-18 00:57:29,973 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DB98F868A5644A0AB164B88DBF74EBFB, mem on start (mb), free: 312, total: 357, max: 672
2016-05-18 00:57:29,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:29,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:29,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:29,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:29,973 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CE9AE8AAB6D343ACA24F0F41283F533A:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [BCB8ACC9695A43EEA14958B37DE41440:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:29,974 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:29,974 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:29,974 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:29,974 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CE9AE8AAB6D343ACA24F0F41283F533A:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [BCB8ACC9695A43EEA14958B37DE41440:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:29,974 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:29,975 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CE9AE8AAB6D343ACA24F0F41283F533A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:29,975 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:29,975 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:29,975 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CE9AE8AAB6D343ACA24F0F41283F533A
2016-05-18 00:57:29,976 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:29,976 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533049735_0001_1_00_000000_0
2016-05-18 00:57:29,978 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:29,979 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - CE9AE8AAB6D343ACA24F0F41283F533A initialized RecordReader from event
2016-05-18 00:57:29,981 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:29,981 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: CE9AE8AAB6D343ACA24F0F41283F533A
2016-05-18 00:57:29,981 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: F1279AA585B742A28B1FEEA11380EE31
2016-05-18 00:57:29,981 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] F1279AA585B742A28B1FEEA11380EE31
2016-05-18 00:57:29,982 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:29,985 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:29,985 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - BCB8ACC9695A43EEA14958B37DE41440 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@8c896ca, comparator=cascading.tuple.hadoop.util.TupleComparator@2923ac47, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:29,986 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for BCB8ACC9695A43EEA14958B37DE41440: , UsingHashComparator=false
2016-05-18 00:57:30,000 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:30,001 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:30,001 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for BCB8ACC9695A43EEA14958B37DE41440: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:30,001 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - BCB8ACC9695A43EEA14958B37DE41440: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:30,002 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:30,002 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DB98F868A5644A0AB164B88DBF74EBFB, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:30,006 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533049735_0001_1_00_000000_0
2016-05-18 00:57:30,006 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DB98F868A5644A0AB164B88DBF74EBFB, mem on close (mb), free: 210, total: 357, max: 672
2016-05-18 00:57:30,006 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - BCB8ACC9695A43EEA14958B37DE41440: Starting flush of map output
2016-05-18 00:57:30,006 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - BCB8ACC9695A43EEA14958B37DE41440: Span0.length = 20, perItem = 6
2016-05-18 00:57:30,007 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - BCB8ACC9695A43EEA14958B37DE41440: done sorting span=0, length=20, time=0
2016-05-18 00:57:30,007 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - BCB8ACC9695A43EEA14958B37DE41440: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:30,009 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - BCB8ACC9695A43EEA14958B37DE41440: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001_wd/localmode-local-dir/output/attempt_1463533049735_0001_1_00_000000_0_10057_0/file.out
2016-05-18 00:57:30,013 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:30 Completed running task attempt: attempt_1463533049735_0001_1_00_000000_0
2016-05-18 00:57:30,014 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533049735_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:30,014 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533049735_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:30,014 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533049735_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533050002, Process_Duration=3, Process_End_Time=1463533050005, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:30,015 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533049735_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:30,015 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:30,016 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DB98F868A5644A0AB164B88DBF74EBFB, taskAttemptId=attempt_1463533049735_0001_1_00_000000_0, creationTime=1463533049943, allocationTime=1463533049950, startTime=1463533049952, finishTime=1463533050015, timeTaken=63, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:30,016 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533049735_0001_1_00_000000_0
2016-05-18 00:57:30,016 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:30,017 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:TASK_FINISHED]: vertexName=DB98F868A5644A0AB164B88DBF74EBFB, taskId=task_1463533049735_0001_1_00_000000, startTime=1463533049952, finishTime=1463533050016, timeTaken=64, status=SUCCEEDED, successfulAttemptID=attempt_1463533049735_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533050002, Process_Duration=3, Process_End_Time=1463533050005, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:30,017 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,018 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:30,018 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:30,018 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533049735_0001_00_000001
2016-05-18 00:57:30,019 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:VERTEX_FINISHED]: vertexName=DB98F868A5644A0AB164B88DBF74EBFB, vertexId=vertex_1463533049735_0001_1_00, initRequestedTime=1463533049916, initedTime=1463533049937, startRequestedTime=1463533049918, startedTime=1463533049938, finishTime=1463533050018, timeTaken=80, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533050002, Process_Duration=3, Process_End_Time=1463533050005, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533049952, firstTasksToStart=[ task_1463533049735_0001_1_00_000000 ], lastTaskFinishTime=1463533050015, lastTasksToFinish=[ task_1463533049735_0001_1_00_000000 ], minTaskDuration=63, maxTaskDuration=63, avgTaskDuration=63.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533049735_0001_1_00_000000 ], longestDurationTasks=[ task_1463533049735_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:30,020 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:30,020 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533049735_0001_00_000001, stoppedTime=1463533050020, exitStatus=0
2016-05-18 00:57:30,021 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440] attempt: attempt_1463533049735_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:30,021 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: BCB8ACC9695A43EEA14958B37DE41440
2016-05-18 00:57:30,021 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=DB98F868A5644A0AB164B88DBF74EBFB, destinationVertexName=BCB8ACC9695A43EEA14958B37DE41440. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:30,021 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]
2016-05-18 00:57:30,022 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:30,022 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:30,023 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533049735_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:30,023 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:TASK_STARTED]: vertexName=BCB8ACC9695A43EEA14958B37DE41440, taskId=task_1463533049735_0001_1_01_000000, scheduledTime=1463533050023, launchTime=1463533050023
2016-05-18 00:57:30,024 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]
2016-05-18 00:57:30,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533049735_0001_00_000002, launchTime=1463533050025
2016-05-18 00:57:30,026 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533049735_0001_00_000002
2016-05-18 00:57:30,028 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533049735_0001_1_01_000000_0] started. Is using containerId: [container_1463533049735_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:30,028 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BCB8ACC9695A43EEA14958B37DE41440, taskAttemptId=attempt_1463533049735_0001_1_01_000000_0, startTime=1463533050027, containerId=container_1463533049735_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:30,030 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533049735_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533049735_0001_1_01_000000_0
2016-05-18 00:57:30,030 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533049735_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:30,030 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533049735_0001_00_000001: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:30,030 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533049735_0001_00_000001, Exiting
2016-05-18 00:57:30,030 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533049735_0001_00_000001
2016-05-18 00:57:30,031 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533049735_0001_00_000001
2016-05-18 00:57:30 Starting to run new task attempt: attempt_1463533049735_0001_1_01_000000_0
2016-05-18 00:57:30,031 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533049735_0001_1_01_000000_0
2016-05-18 00:57:30,031 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,032 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:30,032 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F] every/(1/1) ...yplatform/-1_UNKNOWN_SWAP, VertexName: BCB8ACC9695A43EEA14958B37DE41440, VertexParallelism: 1, TaskAttemptID:attempt_1463533049735_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DB98F868A5644A0AB164B88DBF74EBFB, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8EAD7471BB924E35B52870B86F8A1599, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:30,032 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:30,036 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533049735_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:30,036 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533049735_0001_00_000001 completed successfully
2016-05-18 00:57:30,037 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533049735_0001_1_01_000000_0
2016-05-18 00:57:30,040 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533049735_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]
2016-05-18 00:57:30,041 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533049735_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:30,041 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:30,044 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:30,056 INFO output.MROutput (MROutput.java:initialize(421)) - 8EAD7471BB924E35B52870B86F8A1599: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:30,062 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BCB8ACC9695A43EEA14958B37DE41440, ordinal: 1
2016-05-18 00:57:30,062 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BCB8ACC9695A43EEA14958B37DE41440, mem on start (mb), free: 206, total: 357, max: 672
2016-05-18 00:57:30,062 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:30,062 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:30,062 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:30,062 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:30,063 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DB98F868A5644A0AB164B88DBF74EBFB:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [8EAD7471BB924E35B52870B86F8A1599:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:30,063 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:30,063 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:30,063 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:30,064 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DB98F868A5644A0AB164B88DBF74EBFB:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [8EAD7471BB924E35B52870B86F8A1599:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:30,064 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:30,064 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - DB98F868A5644A0AB164B88DBF74EBFB: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:30,064 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - DB98F868A5644A0AB164B88DBF74EBFB: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:30,065 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DB98F868A5644A0AB164B88DBF74EBFB being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:30,065 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:30,065 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:30,066 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: DB98F868A5644A0AB164B88DBF74EBFB with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:30,066 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DB98F868A5644A0AB164B88DBF74EBFB
2016-05-18 00:57:30,068 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:30,068 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533049735_0001_1_01_000000_0
2016-05-18 00:57:30,071 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DB98F868A5644A0AB164B88DBF74EBFB: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:30,073 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:30,073 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533049735_0001_1_00_000000_0_10057} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533050073, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:30,073 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : DB98F868A5644A0AB164B88DBF74EBFB
2016-05-18 00:57:30,073 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: F1279AA585B742A28B1FEEA11380EE31
2016-05-18 00:57:30,074 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_SWAP"], id: 8EAD7471BB924E35B52870B86F8A1599
2016-05-18 00:57:30,074 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:30,074 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_SWAP"]
2016-05-18 00:57:30,074 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] F1279AA585B742A28B1FEEA11380EE31, for 1 inputs
2016-05-18 00:57:30,074 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: DB98F868A5644A0AB164B88DBF74EBFB, wasInterrupted=false
2016-05-18 00:57:30,075 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:30,076 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:30,077 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:30,077 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001_wd/localmode-local-dir/output/attempt_1463533049735_0001_1_00_000000_0_10057/file.out, len=181, isLocal=true
2016-05-18 00:57:30,077 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:30,077 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:30,077 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:30,077 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:30,078 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : DB98F868A5644A0AB164B88DBF74EBFB
2016-05-18 00:57:30,078 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - DB98F868A5644A0AB164B88DBF74EBFB: Shuffle Runner thread complete
2016-05-18 00:57:30,078 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BCB8ACC9695A43EEA14958B37DE41440, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:30,079 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:30,079 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - DB98F868A5644A0AB164B88DBF74EBFB: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:30,085 INFO output.MROutput (MROutput.java:close(526)) - 8EAD7471BB924E35B52870B86F8A1599 closed
2016-05-18 00:57:30,086 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533049735_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:30,087 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533049735_0001_1_01_000000_0
2016-05-18 00:57:30,087 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BCB8ACC9695A43EEA14958B37DE41440, mem on close (mb), free: 202, total: 357, max: 672
2016-05-18 00:57:30,087 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: DB98F868A5644A0AB164B88DBF74EBFB
2016-05-18 00:57:30,087 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DB98F868A5644A0AB164B88DBF74EBFB: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:30,087 INFO output.MROutput (MROutput.java:close(526)) - 8EAD7471BB924E35B52870B86F8A1599 closed
2016-05-18 00:57:30,087 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:30 Completed running task attempt: attempt_1463533049735_0001_1_01_000000_0
2016-05-18 00:57:30,088 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533049735_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:30,088 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533049735_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:30,088 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533049735_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533050078, Process_Duration=9, Process_End_Time=1463533050087, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:30,090 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533049735_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:30,090 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BCB8ACC9695A43EEA14958B37DE41440, taskAttemptId=attempt_1463533049735_0001_1_01_000000_0, creationTime=1463533050023, allocationTime=1463533050025, startTime=1463533050027, finishTime=1463533050090, timeTaken=63, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:30,090 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:30,090 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:30,090 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,090 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533049735_0001_1_01_000000_0
2016-05-18 00:57:30,091 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533049735_0001_00_000002
2016-05-18 00:57:30,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:TASK_FINISHED]: vertexName=BCB8ACC9695A43EEA14958B37DE41440, taskId=task_1463533049735_0001_1_01_000000, startTime=1463533050027, finishTime=1463533050091, timeTaken=64, status=SUCCEEDED, successfulAttemptID=attempt_1463533049735_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533050078, Process_Duration=9, Process_End_Time=1463533050087, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:30,092 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:30,092 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440]
2016-05-18 00:57:30,093 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:VERTEX_FINISHED]: vertexName=BCB8ACC9695A43EEA14958B37DE41440, vertexId=vertex_1463533049735_0001_1_01, initRequestedTime=1463533049918, initedTime=1463533049925, startRequestedTime=1463533049939, startedTime=1463533049939, finishTime=1463533050092, timeTaken=153, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533050078, Process_Duration=9, Process_End_Time=1463533050087, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533050027, firstTasksToStart=[ task_1463533049735_0001_1_01_000000 ], lastTaskFinishTime=1463533050090, lastTasksToFinish=[ task_1463533049735_0001_1_01_000000 ], minTaskDuration=63, maxTaskDuration=63, avgTaskDuration=63.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533049735_0001_1_01_000000 ], longestDurationTasks=[ task_1463533049735_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:30,094 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:30,094 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533049735_0001_00_000002, stoppedTime=1463533050094, exitStatus=0
2016-05-18 00:57:30,094 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:30,094 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:30,094 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533049735_0001_1_00 [DB98F868A5644A0AB164B88DBF74EBFB]
2016-05-18 00:57:30,095 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533049735_0001_1
2016-05-18 00:57:30,095 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533049735_0001_1
2016-05-18 00:57:30,096 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 8EAD7471BB924E35B52870B86F8A1599 for vertex: vertex_1463533049735_0001_1_01 [BCB8ACC9695A43EEA14958B37DE41440], outputName: 8EAD7471BB924E35B52870B86F8A1599
2016-05-18 00:57:30,100 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533049735_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:30,100 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533049735_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:30,102 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533049735_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:30,102 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533049735_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:30,102 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533049735_0001_00_000002, Exiting
2016-05-18 00:57:30,102 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533049735_0001_00_000002
2016-05-18 00:57:30,102 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533049735_0001_00_000002
2016-05-18 00:57:30,103 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533049735_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:30,103 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533049735_0001_00_000002 completed successfully
2016-05-18 00:57:30,104 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:8EAD7471BB924E35B52870B86F8A1599 of vertex/vertexGroup:BCB8ACC9695A43EEA14958B37DE41440 isVertexGroupOutput:false
2016-05-18 00:57:30,104 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:30,106 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533049735_0001_1, queueSize=0
2016-05-18 00:57:30,107 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533049735_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533049735_0001_1, startTime=1463533049916, finishTime=1463533050104, timeTaken=188, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=410, AM_GC_TIME_MILLIS=7, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066100080, Process_Duration=12, Process_End_Time=2927066100092, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:30,107 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533049735_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:30,107 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533049735_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:30 Completed Dag: dag_1463533049735_0001_1
2016-05-18 00:57:30,108 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533049735_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:30,108 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533049735_0001_1_post
2016-05-18 00:57:30,108 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,108 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:30,108 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:30,108 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F] every/(1/1) ...yplatform/-1_UNKNOWN_SWAP, with id=dag_1463533049735_0001_1
2016-05-18 00:57:30,108 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F] every/(1/1) ...yplatform/-1_UNKNOWN_SWAP, with id=dag_1463533049735_0001_1
2016-05-18 00:57:30,117 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...yplatform/-1_UNKNOWN_SWAP, applicationId=application_1463533049735_0001
2016-05-18 00:57:30,119 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:30,119 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:30,120 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:30,120 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:30,120 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:30,120 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:30,121 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:30,121 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:30,121 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:30,121 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:30,122 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:30,122 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:30,122 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:30,123 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:30,128 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/896F595D978D4BB8BE5C52A22AE4C3D2/C04497D3261A48469B6E18150B1B8D2F/.tez/application_1463533049735_0001
2016-05-18 00:57:30,129 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_UNKNOWN_SWAP/_temporary
2016-05-18 00:57:30,129 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.425
2016-05-18 00:57:30,169 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:30,169 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:30,217 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_UNKNOWN_SWAP
2016-05-18 00:57:30,243 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:30,244 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:30,282 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.009
2016-05-18 00:57:30,293 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.020
2016-05-18 00:57:30,293 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:30,294 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:30,294 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:30,314 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:30,320 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_-1
2016-05-18 00:57:30,375 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:30,375 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:30,375 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_-1"]
2016-05-18 00:57:30,376 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:30,376 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:30,376 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:30,376 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...mblyplatform/-1_VALUES_-1
2016-05-18 00:57:30,377 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:30,390 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:30,394 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:30,394 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:30,395 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:30,403 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001 doesn't exist and is created
2016-05-18 00:57:30,418 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:30,418 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:30,419 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001_wd
2016-05-18 00:57:30,428 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533050394_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:30,428 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:30,429 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:30,430 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:30,430 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:30,430 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:30,430 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:30,431 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:30,433 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:30,433 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:30,433 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:30,433 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:30,433 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:30,433 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:30,433 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:30,433 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:30,434 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:30,434 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:30,434 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:30,434 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:30,436 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:30,437 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:30,437 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:30,437 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001_wd/localmode-log-dir
2016-05-18 00:57:30,437 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001_wd/localmode-log-dir/history.txt.appattempt_1463533050394_0001_000000, maxErrors=10
2016-05-18 00:57:30,437 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:30,438 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533050394_0001_000000, appSubmitTime=1463533050428, launchTime=1463533050428
2016-05-18 00:57:30,444 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:30,450 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:49832
2016-05-18 00:57:30,450 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533050394
2016-05-18 00:57:30,452 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533050394_0001_000000, startTime=1463533050452
2016-05-18 00:57:30,452 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:30,518 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:30,518 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:30,519 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...mblyplatform/-1_VALUES_-1, applicationId=application_1463533050394_0001, dagName=[E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B] every/(1/1) ...mblyplatform/-1_VALUES_-1
2016-05-18 00:57:30,535 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B] every/(1/1) ...mblyplatform/-1_VALUES_-1
2016-05-18 00:57:30,536 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:30,536 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533050394_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001_wd/localmode-log-dir/dag_1463533050394_0001_1.dot
2016-05-18 00:57:30,537 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001_wd/localmode-log-dir/dag_1463533050394_0001_1-tez-dag.pb.txt
2016-05-18 00:57:30,553 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533050394_0001_1
2016-05-18 00:57:30,554 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,554 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B] every/(1/1) ...mblyplatform/-1_VALUES_-1
2016-05-18 00:57:30 Running Dag: dag_1463533050394_0001_1
2016-05-18 00:57:30,563 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533050394_0001_1, submitTime=1463533050535
2016-05-18 00:57:30,564 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001/tez-conf.pb" } size: 36913 timestamp: 1463533050000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:30,565 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,565 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:30,565 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001/tez-conf.pb" } size: 36913 timestamp: 1463533050000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:30,566 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,566 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:30,566 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:30,567 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533050394_0001_1, initTime=1463533050566
2016-05-18 00:57:30,567 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533050394_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:30,567 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:30,568 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:DAG_STARTED]: dagID=dag_1463533050394_0001_1, startTime=1463533050568
2016-05-18 00:57:30,568 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533050394_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:30,568 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: CC49ACE008C04FFE8D3BF78E1E9308A0 : {79D88F5085234EA9B393E2778B1DBBF6={InputName=79D88F5085234EA9B393E2778B1DBBF6}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:30,568 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 79D88F5085234EA9B393E2778B1DBBF6, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:30,568 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,569 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,569 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,569 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,569 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:30,569 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...mblyplatform/-1_VALUES_-1, applicationId=application_1463533050394_0001, dagName=[E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B] every/(1/1) ...mblyplatform/-1_VALUES_-1
2016-05-18 00:57:30,570 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,573 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:30,573 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,574 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,574 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 79D88F5085234EA9B393E2778B1DBBF6 on vertex vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,575 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533050394_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:30,575 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2], numAdditionalOutputs=1
2016-05-18 00:57:30,575 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=B16BA05D152E4B308386B7A5943A0DF0, vertex=vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:30,577 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 79D88F5085234EA9B393E2778B1DBBF6 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:30,577 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for EFA5E8A264CF40B3945D1EB2B8950CE2:B16BA05D152E4B308386B7A5943A0DF0 using oldmapred API
2016-05-18 00:57:30,578 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=B16BA05D152E4B308386B7A5943A0DF0, vertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:30,579 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:30,580 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:30,580 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:30,581 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:30,581 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 79D88F5085234EA9B393E2778B1DBBF6 on vertex vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,582 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, vertexId=vertex_1463533050394_0001_1_01, initRequestedTime=1463533050570, initedTime=1463533050575, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:30,582 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:30,583 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:30,584 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {79D88F5085234EA9B393E2778B1DBBF6=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:30,584 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] parallelism set to 1
2016-05-18 00:57:30,584 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:30,585 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CC49ACE008C04FFE8D3BF78E1E9308A0, vertexId=vertex_1463533050394_0001_1_00, initRequestedTime=1463533050568, initedTime=1463533050585, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:30,585 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] with distanceFromRoot: 0
2016-05-18 00:57:30,585 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:30,585 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533050394_0001_1_00, startRequestedTime=1463533050570, startedTime=1463533050585
2016-05-18 00:57:30,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533050394_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:30,586 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:30,586 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533050394_0001_1_00 for vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] numStartedSources: 1 numSources: 1
2016-05-18 00:57:30,586 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] with distanceFromRoot: 1
2016-05-18 00:57:30,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533050394_0001_1_01, startRequestedTime=1463533050586, startedTime=1463533050586
2016-05-18 00:57:30,587 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:30,588 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in CC49ACE008C04FFE8D3BF78E1E9308A0
2016-05-18 00:57:30,589 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,589 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: EFA5E8A264CF40B3945D1EB2B8950CE2 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:30,590 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:TASK_STARTED]: vertexName=CC49ACE008C04FFE8D3BF78E1E9308A0, taskId=task_1463533050394_0001_1_00_000000, scheduledTime=1463533050590, launchTime=1463533050590
2016-05-18 00:57:30,591 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,592 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: CC49ACE008C04FFE8D3BF78E1E9308A0 in vertex: EFA5E8A264CF40B3945D1EB2B8950CE2 numBipartiteSourceTasks: 1
2016-05-18 00:57:30,592 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:30,594 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533050394_0001_00_000001, launchTime=1463533050594
2016-05-18 00:57:30,594 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533050394_0001, with dag id: dag_1463533050394_0001_1
2016-05-18 00:57:30,597 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533050394_0001_00_000001
2016-05-18 00:57:30,598 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533050394_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533050394_0001_1_00_000000_0
2016-05-18 00:57:30,598 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533050394_0001_1_00_000000_0] started. Is using containerId: [container_1463533050394_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:30 Starting to run new task attempt: attempt_1463533050394_0001_1_00_000000_0
2016-05-18 00:57:30,598 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CC49ACE008C04FFE8D3BF78E1E9308A0, taskAttemptId=attempt_1463533050394_0001_1_00_000000_0, startTime=1463533050597, containerId=container_1463533050394_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:30,598 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533050394_0001_1_00_000000_0
2016-05-18 00:57:30,598 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,598 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:30,598 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B] every/(1/1) ...mblyplatform/-1_VALUES_-1, VertexName: CC49ACE008C04FFE8D3BF78E1E9308A0, VertexParallelism: 1, TaskAttemptID:attempt_1463533050394_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=79D88F5085234EA9B393E2778B1DBBF6, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:30,599 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:30,599 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533050394_0001_1_00_000000_0
2016-05-18 00:57:30,600 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533050394_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:30,601 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533050394_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:30,604 INFO input.MRInput (MRInput.java:initialize(440)) - 79D88F5085234EA9B393E2778B1DBBF6 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:30,604 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 79D88F5085234EA9B393E2778B1DBBF6
2016-05-18 00:57:30,608 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:30,614 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CC49ACE008C04FFE8D3BF78E1E9308A0, ordinal: 0
2016-05-18 00:57:30,614 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CC49ACE008C04FFE8D3BF78E1E9308A0, mem on start (mb), free: 316, total: 365, max: 672
2016-05-18 00:57:30,614 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:30,614 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:30,614 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:30,614 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:30,614 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[79D88F5085234EA9B393E2778B1DBBF6:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [EFA5E8A264CF40B3945D1EB2B8950CE2:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:30,615 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:30,615 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:30,615 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:30,615 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[79D88F5085234EA9B393E2778B1DBBF6:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [EFA5E8A264CF40B3945D1EB2B8950CE2:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:30,615 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:30,616 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 79D88F5085234EA9B393E2778B1DBBF6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:30,616 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 79D88F5085234EA9B393E2778B1DBBF6
2016-05-18 00:57:30,616 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:30,616 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:30,616 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:30,617 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533050394_0001_1_00_000000_0
2016-05-18 00:57:30,618 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:30,618 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 79D88F5085234EA9B393E2778B1DBBF6 initialized RecordReader from event
2016-05-18 00:57:30,620 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:30,620 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 79D88F5085234EA9B393E2778B1DBBF6
2016-05-18 00:57:30,620 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 4E7DAF29B1484FBEBCC2496893036756
2016-05-18 00:57:30,621 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 4E7DAF29B1484FBEBCC2496893036756
2016-05-18 00:57:30,622 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:30,624 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:30,624 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - EFA5E8A264CF40B3945D1EB2B8950CE2 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@201ed4a, comparator=cascading.tuple.hadoop.util.TupleComparator@4c81964d, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:30,625 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for EFA5E8A264CF40B3945D1EB2B8950CE2: , UsingHashComparator=false
2016-05-18 00:57:30,638 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:30,638 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:30,638 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for EFA5E8A264CF40B3945D1EB2B8950CE2: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:30,638 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - EFA5E8A264CF40B3945D1EB2B8950CE2: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:30,638 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:30,639 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CC49ACE008C04FFE8D3BF78E1E9308A0, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:30,643 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533050394_0001_1_00_000000_0
2016-05-18 00:57:30,643 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CC49ACE008C04FFE8D3BF78E1E9308A0, mem on close (mb), free: 214, total: 365, max: 672
2016-05-18 00:57:30,643 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - EFA5E8A264CF40B3945D1EB2B8950CE2: Starting flush of map output
2016-05-18 00:57:30,643 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - EFA5E8A264CF40B3945D1EB2B8950CE2: Span0.length = 20, perItem = 6
2016-05-18 00:57:30,644 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - EFA5E8A264CF40B3945D1EB2B8950CE2: done sorting span=0, length=20, time=1
2016-05-18 00:57:30,644 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - EFA5E8A264CF40B3945D1EB2B8950CE2: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:30,645 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - EFA5E8A264CF40B3945D1EB2B8950CE2: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001_wd/localmode-local-dir/output/attempt_1463533050394_0001_1_00_000000_0_10063_0/file.out
2016-05-18 00:57:30,649 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:30 Completed running task attempt: attempt_1463533050394_0001_1_00_000000_0
2016-05-18 00:57:30,649 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533050394_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:30,650 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533050394_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:30,650 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533050394_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533050639, Process_Duration=4, Process_End_Time=1463533050643, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:30,651 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533050394_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:30,651 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:30,651 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CC49ACE008C04FFE8D3BF78E1E9308A0, taskAttemptId=attempt_1463533050394_0001_1_00_000000_0, creationTime=1463533050591, allocationTime=1463533050593, startTime=1463533050597, finishTime=1463533050651, timeTaken=54, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:30,651 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:30,651 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,652 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533050394_0001_00_000001
2016-05-18 00:57:30,652 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533050394_0001_1_00_000000_0
2016-05-18 00:57:30,652 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:TASK_FINISHED]: vertexName=CC49ACE008C04FFE8D3BF78E1E9308A0, taskId=task_1463533050394_0001_1_00_000000, startTime=1463533050597, finishTime=1463533050652, timeTaken=55, status=SUCCEEDED, successfulAttemptID=attempt_1463533050394_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533050639, Process_Duration=4, Process_End_Time=1463533050643, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:30,653 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:30,653 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0]
2016-05-18 00:57:30,655 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:VERTEX_FINISHED]: vertexName=CC49ACE008C04FFE8D3BF78E1E9308A0, vertexId=vertex_1463533050394_0001_1_00, initRequestedTime=1463533050568, initedTime=1463533050585, startRequestedTime=1463533050570, startedTime=1463533050585, finishTime=1463533050654, timeTaken=69, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533050639, Process_Duration=4, Process_End_Time=1463533050643, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533050597, firstTasksToStart=[ task_1463533050394_0001_1_00_000000 ], lastTaskFinishTime=1463533050651, lastTasksToFinish=[ task_1463533050394_0001_1_00_000000 ], minTaskDuration=54, maxTaskDuration=54, avgTaskDuration=54.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533050394_0001_1_00_000000 ], longestDurationTasks=[ task_1463533050394_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:30,655 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:30,655 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533050394_0001_00_000001, stoppedTime=1463533050655, exitStatus=0
2016-05-18 00:57:30,656 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] attempt: attempt_1463533050394_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:30,656 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EFA5E8A264CF40B3945D1EB2B8950CE2
2016-05-18 00:57:30,656 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=CC49ACE008C04FFE8D3BF78E1E9308A0, destinationVertexName=EFA5E8A264CF40B3945D1EB2B8950CE2. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:30,657 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,657 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533050394_0001_1_00 [CC49ACE008C04FFE8D3BF78E1E9308A0] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:30,657 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:30,657 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533050394_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:30,658 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:TASK_STARTED]: vertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, taskId=task_1463533050394_0001_1_01_000000, scheduledTime=1463533050657, launchTime=1463533050657
2016-05-18 00:57:30,658 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,660 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533050394_0001_00_000002, launchTime=1463533050660
2016-05-18 00:57:30,661 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533050394_0001_00_000002
2016-05-18 00:57:30,662 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533050394_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533050394_0001_1_01_000000_0
2016-05-18 00:57:30,662 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533050394_0001_1_01_000000_0] started. Is using containerId: [container_1463533050394_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:30,662 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, taskAttemptId=attempt_1463533050394_0001_1_01_000000_0, startTime=1463533050661, containerId=container_1463533050394_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:30 Starting to run new task attempt: attempt_1463533050394_0001_1_01_000000_0
2016-05-18 00:57:30,662 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533050394_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:30,662 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533050394_0001_1_01_000000_0
2016-05-18 00:57:30,662 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,662 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533050394_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:30,662 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:30,663 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533050394_0001_00_000001, Exiting
2016-05-18 00:57:30,663 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533050394_0001_00_000001
2016-05-18 00:57:30,663 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533050394_0001_00_000001
2016-05-18 00:57:30,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B] every/(1/1) ...mblyplatform/-1_VALUES_-1, VertexName: EFA5E8A264CF40B3945D1EB2B8950CE2, VertexParallelism: 1, TaskAttemptID:attempt_1463533050394_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CC49ACE008C04FFE8D3BF78E1E9308A0, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B16BA05D152E4B308386B7A5943A0DF0, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:30,664 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:30,665 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533050394_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:30,665 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533050394_0001_1_01_000000_0
2016-05-18 00:57:30,666 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533050394_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,666 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533050394_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:30,666 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533050394_0001_00_000001 completed successfully
2016-05-18 00:57:30,669 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:30,673 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:30,679 INFO output.MROutput (MROutput.java:initialize(421)) - B16BA05D152E4B308386B7A5943A0DF0: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:30,681 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EFA5E8A264CF40B3945D1EB2B8950CE2, ordinal: 1
2016-05-18 00:57:30,681 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EFA5E8A264CF40B3945D1EB2B8950CE2, mem on start (mb), free: 209, total: 365, max: 672
2016-05-18 00:57:30,681 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:30,681 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:30,681 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:30,681 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:30,681 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CC49ACE008C04FFE8D3BF78E1E9308A0:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [B16BA05D152E4B308386B7A5943A0DF0:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:30,681 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:30,682 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:30,682 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:30,682 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CC49ACE008C04FFE8D3BF78E1E9308A0:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [B16BA05D152E4B308386B7A5943A0DF0:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:30,682 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:30,682 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CC49ACE008C04FFE8D3BF78E1E9308A0 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:30,682 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - CC49ACE008C04FFE8D3BF78E1E9308A0: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:30,683 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - CC49ACE008C04FFE8D3BF78E1E9308A0: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:30,684 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: CC49ACE008C04FFE8D3BF78E1E9308A0 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:30,685 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CC49ACE008C04FFE8D3BF78E1E9308A0
2016-05-18 00:57:30,685 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:30,685 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:30,685 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:30,688 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533050394_0001_1_01_000000_0
2016-05-18 00:57:30,689 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CC49ACE008C04FFE8D3BF78E1E9308A0: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:30,691 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533050394_0001_1_00_000000_0_10063} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533050690, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:30,691 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:30,691 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : CC49ACE008C04FFE8D3BF78E1E9308A0
2016-05-18 00:57:30,692 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 4E7DAF29B1484FBEBCC2496893036756
2016-05-18 00:57:30,692 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_-1"], id: B16BA05D152E4B308386B7A5943A0DF0
2016-05-18 00:57:30,692 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_-1"]
2016-05-18 00:57:30,693 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 4E7DAF29B1484FBEBCC2496893036756, for 1 inputs
2016-05-18 00:57:30,692 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:30,693 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: CC49ACE008C04FFE8D3BF78E1E9308A0, wasInterrupted=false
2016-05-18 00:57:30,694 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:30,694 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:30,696 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:30,696 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001_wd/localmode-local-dir/output/attempt_1463533050394_0001_1_00_000000_0_10063/file.out, len=181, isLocal=true
2016-05-18 00:57:30,696 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:30,696 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:30,696 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:30,697 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:30,697 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : CC49ACE008C04FFE8D3BF78E1E9308A0
2016-05-18 00:57:30,697 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - CC49ACE008C04FFE8D3BF78E1E9308A0: Shuffle Runner thread complete
2016-05-18 00:57:30,697 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EFA5E8A264CF40B3945D1EB2B8950CE2, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:30,698 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:30,698 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - CC49ACE008C04FFE8D3BF78E1E9308A0: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:30,701 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:30,702 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:30,704 INFO output.MROutput (MROutput.java:close(526)) - B16BA05D152E4B308386B7A5943A0DF0 closed
2016-05-18 00:57:30,704 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533050394_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:30,705 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533050394_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:30,705 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533050394_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533050697, Process_Duration=8, Process_End_Time=1463533050705, Read_Duration=1, Tuples_Read=1]]
2016-05-18 00:57:30,705 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:30,706 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EFA5E8A264CF40B3945D1EB2B8950CE2, mem on close (mb), free: 206, total: 365, max: 672
2016-05-18 00:57:30,706 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=EFA5E8A264CF40B3945D1EB2B8950CE2, index=1, interruptedStatus=false
2016-05-18 00:57:30,706 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: CC49ACE008C04FFE8D3BF78E1E9308A0
2016-05-18 00:57:30,706 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CC49ACE008C04FFE8D3BF78E1E9308A0: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:30,706 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=EFA5E8A264CF40B3945D1EB2B8950CE2, sourceVertex=CC49ACE008C04FFE8D3BF78E1E9308A0, interruptedStatus=true
2016-05-18 00:57:30,706 INFO output.MROutput (MROutput.java:close(526)) - B16BA05D152E4B308386B7A5943A0DF0 closed
2016-05-18 00:57:30,707 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=EFA5E8A264CF40B3945D1EB2B8950CE2, sourceVertex=B16BA05D152E4B308386B7A5943A0DF0, interruptedStatus=true
2016-05-18 00:57:30,708 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533050394_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:30,708 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:30,708 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533050394_0001_00_000002
2016-05-18 00:57:30,708 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, taskAttemptId=attempt_1463533050394_0001_1_01_000000_0, creationTime=1463533050658, allocationTime=1463533050659, startTime=1463533050661, finishTime=1463533050708, timeTaken=47, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533050394_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533050697, Process_Duration=8, Process_End_Time=1463533050705, Read_Duration=1, Tuples_Read=1
2016-05-18 00:57:30,709 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533050394_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:30,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:TASK_FINISHED]: vertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, taskId=task_1463533050394_0001_1_01_000000, startTime=1463533050661, finishTime=1463533050709, timeTaken=48, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533050394_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:30,710 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] because task failed: task_1463533050394_0001_1_01_000000
2016-05-18 00:57:30,710 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:30,710 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:30,710 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:30,710 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,708 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533050394_0001_00_000002
2016-05-18 00:57:30,710 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=B16BA05D152E4B308386B7A5943A0DF0, vertexId=vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,710 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533050394_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:30,711 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533050394_0001_00_000002 completed but with errors
2016-05-18 00:57:30,713 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:VERTEX_FINISHED]: vertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, vertexId=vertex_1463533050394_0001_1_01, initRequestedTime=1463533050570, initedTime=1463533050575, startRequestedTime=1463533050586, startedTime=1463533050586, finishTime=1463533050712, timeTaken=126, status=FAILED, diagnostics=Task failed, taskId=task_1463533050394_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533050394_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533050661, firstTasksToStart=[ task_1463533050394_0001_1_01_000000 ], lastTaskFinishTime=1463533050708, lastTasksToFinish=[ task_1463533050394_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:30,713 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:30,714 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533050394_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:30,714 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533050394_0001_00_000002, stoppedTime=1463533050714, exitStatus=1
2016-05-18 00:57:30,714 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533050394_0001_1_01_000000_0 failed: true container: container_1463533050394_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:30,714 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:30,714 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:30,714 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:30,717 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533050394_0001_1, queueSize=0
2016-05-18 00:57:30,717 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050394_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533050394_0001_1, startTime=1463533050567, finishTime=1463533050716, timeTaken=149, status=FAILED, diagnostics=Vertex failed, vertexName=EFA5E8A264CF40B3945D1EB2B8950CE2, vertexId=vertex_1463533050394_0001_1_01, diagnostics=[Task failed, taskId=task_1463533050394_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533050394_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=280, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533050639, Process_Duration=4, Process_End_Time=1463533050643, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:30,718 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533050394_0001_1_01 [EFA5E8A264CF40B3945D1EB2B8950CE2]
2016-05-18 00:57:30,718 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533050394_0001_1 finished with state: FAILED
2016-05-18 00:57:30,718 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533050394_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:30 Completed Dag: dag_1463533050394_0001_1
2016-05-18 00:57:30,719 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533050394_0001_1, dagState=FAILED
2016-05-18 00:57:30,719 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533050394_0001_1_post
2016-05-18 00:57:30,719 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:30,719 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:30,719 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:30,719 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B] every/(1/1) ...mblyplatform/-1_VALUES_-1, with id=dag_1463533050394_0001_1
2016-05-18 00:57:30,719 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B] every/(1/1) ...mblyplatform/-1_VALUES_-1, with id=dag_1463533050394_0001_1
2016-05-18 00:57:30,727 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...mblyplatform/-1_VALUES_-1, applicationId=application_1463533050394_0001
2016-05-18 00:57:30,729 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:30,729 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:30,729 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:30,729 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:30,729 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:30,730 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:30,730 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:30,731 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:30,731 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:30,731 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:30,731 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:30,731 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:30,732 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:30,732 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:30,735 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E01F14C68BDC44639E6703A43A4BFAF3/C9366AC9653D40F5A4B132766CDC4A9B/.tez/application_1463533050394_0001
2016-05-18 00:57:30,735 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:30,736 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...mblyplatform/-1_VALUES_-1
2016-05-18 00:57:30,736 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:30,737 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_-1/_temporary
2016-05-18 00:57:30,737 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.361
2016-05-18 00:57:30,737 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_VALUES_-1
2016-05-18 00:57:30,771 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:30,771 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:30,808 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.004
2016-05-18 00:57:30,811 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.006
2016-05-18 00:57:30,811 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:30,811 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:30,811 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_VALUES_'field'
2016-05-18 00:57:30,837 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:30,837 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:30,879 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:30,886 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.020
2016-05-18 00:57:30,887 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:30,887 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:30,888 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:30,904 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:30,910 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_RESULTS
2016-05-18 00:57:30,933 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:30,933 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:30,961 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:30,961 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:30,961 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_RESULTS"]
2016-05-18 00:57:30,961 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:30,961 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:30,961 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:30,962 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...latform/-1_VALUES_RESULTS
2016-05-18 00:57:30,962 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:30,975 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:30,979 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:30,979 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:30,979 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:30,988 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001 doesn't exist and is created
2016-05-18 00:57:31,003 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:31,003 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:31,004 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001_wd
2016-05-18 00:57:31,013 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533050979_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:31,014 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:31,014 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:31,016 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:31,016 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:31,016 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:31,016 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:31,018 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:31,019 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:31,019 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:31,019 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:31,019 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:31,020 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:31,020 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:31,020 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:31,020 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:31,020 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:31,020 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:31,021 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:31,021 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:31,022 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:31,023 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:31,023 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:31,023 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001_wd/localmode-log-dir
2016-05-18 00:57:31,023 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001_wd/localmode-log-dir/history.txt.appattempt_1463533050979_0001_000000, maxErrors=10
2016-05-18 00:57:31,024 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:31,024 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533050979_0001_000000, appSubmitTime=1463533051013, launchTime=1463533051013
2016-05-18 00:57:31,027 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:31,029 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:47063
2016-05-18 00:57:31,031 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533050979
2016-05-18 00:57:31,032 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533050979_0001_000000, startTime=1463533051032
2016-05-18 00:57:31,033 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:31,103 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:31,104 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:31,104 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...latform/-1_VALUES_RESULTS, applicationId=application_1463533050979_0001, dagName=[70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79] every/(1/1) ...latform/-1_VALUES_RESULTS
2016-05-18 00:57:31,132 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79] every/(1/1) ...latform/-1_VALUES_RESULTS
2016-05-18 00:57:31,133 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:31,133 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533050979_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001_wd/localmode-log-dir/dag_1463533050979_0001_1.dot
2016-05-18 00:57:31,134 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001_wd/localmode-log-dir/dag_1463533050979_0001_1-tez-dag.pb.txt
2016-05-18 00:57:31,140 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533050979_0001_1
2016-05-18 00:57:31,140 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,140 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79] every/(1/1) ...latform/-1_VALUES_RESULTS
2016-05-18 00:57:31 Running Dag: dag_1463533050979_0001_1
2016-05-18 00:57:31,149 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533050979_0001_1, submitTime=1463533051132
2016-05-18 00:57:31,150 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001/tez-conf.pb" } size: 36913 timestamp: 1463533050000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:31,150 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,151 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:31,152 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001/tez-conf.pb" } size: 36913 timestamp: 1463533050000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:31,152 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,152 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:31,152 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:31,153 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533050979_0001_1, initTime=1463533051153
2016-05-18 00:57:31,153 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533050979_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:31,154 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:31,154 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:DAG_STARTED]: dagID=dag_1463533050979_0001_1, startTime=1463533051154
2016-05-18 00:57:31,154 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...latform/-1_VALUES_RESULTS, applicationId=application_1463533050979_0001, dagName=[70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79] every/(1/1) ...latform/-1_VALUES_RESULTS
2016-05-18 00:57:31,154 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533050979_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:31,154 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: AF1520B590E544F8A6EAE66CD734CFE6 : {CB9F63A12D0A4459AF8D887F47CFDC56={InputName=CB9F63A12D0A4459AF8D887F47CFDC56}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:31,155 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: CB9F63A12D0A4459AF8D887F47CFDC56, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:31,155 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,155 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,155 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,155 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,156 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:31,156 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: CB9F63A12D0A4459AF8D887F47CFDC56 on vertex vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,156 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,158 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input CB9F63A12D0A4459AF8D887F47CFDC56 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:31,159 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:31,159 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,160 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,160 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533050979_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:31,160 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5], numAdditionalOutputs=1
2016-05-18 00:57:31,161 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=63ED769EF57F49A388509B50421F2202, vertex=vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:31,162 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:31,162 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:31,162 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:31,162 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:31,163 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: CB9F63A12D0A4459AF8D887F47CFDC56 on vertex vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,163 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for BE08FAEF77DB4358A1A7435A4EF121D5:63ED769EF57F49A388509B50421F2202 using oldmapred API
2016-05-18 00:57:31,164 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=63ED769EF57F49A388509B50421F2202, vertexName=BE08FAEF77DB4358A1A7435A4EF121D5, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:31,164 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BE08FAEF77DB4358A1A7435A4EF121D5, vertexId=vertex_1463533050979_0001_1_01, initRequestedTime=1463533051156, initedTime=1463533051160, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:31,165 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:31,166 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:31,166 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {CB9F63A12D0A4459AF8D887F47CFDC56=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:31,166 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] parallelism set to 1
2016-05-18 00:57:31,166 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:31,168 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AF1520B590E544F8A6EAE66CD734CFE6, vertexId=vertex_1463533050979_0001_1_00, initRequestedTime=1463533051154, initedTime=1463533051167, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:31,169 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] with distanceFromRoot: 0
2016-05-18 00:57:31,169 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:31,169 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533050979_0001_1_00, startRequestedTime=1463533051156, startedTime=1463533051169
2016-05-18 00:57:31,170 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533050979_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:31,170 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:31,170 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533050979_0001_1_00 for vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] numStartedSources: 1 numSources: 1
2016-05-18 00:57:31,171 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] with distanceFromRoot: 1
2016-05-18 00:57:31,170 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in AF1520B590E544F8A6EAE66CD734CFE6
2016-05-18 00:57:31,171 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,171 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533050979_0001_1_01, startRequestedTime=1463533051170, startedTime=1463533051171
2016-05-18 00:57:31,172 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:31,172 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:TASK_STARTED]: vertexName=AF1520B590E544F8A6EAE66CD734CFE6, taskId=task_1463533050979_0001_1_00_000000, scheduledTime=1463533051172, launchTime=1463533051172
2016-05-18 00:57:31,173 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,177 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533050979_0001, with dag id: dag_1463533050979_0001_1
2016-05-18 00:57:31,177 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:31,178 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: BE08FAEF77DB4358A1A7435A4EF121D5 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:31,180 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: AF1520B590E544F8A6EAE66CD734CFE6 in vertex: BE08FAEF77DB4358A1A7435A4EF121D5 numBipartiteSourceTasks: 1
2016-05-18 00:57:31,181 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533050979_0001_00_000001, launchTime=1463533051181
2016-05-18 00:57:31,181 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533050979_0001_00_000001
2016-05-18 00:57:31,182 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533050979_0001_1_00_000000_0] started. Is using containerId: [container_1463533050979_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:31,182 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AF1520B590E544F8A6EAE66CD734CFE6, taskAttemptId=attempt_1463533050979_0001_1_00_000000_0, startTime=1463533051182, containerId=container_1463533050979_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:31,183 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533050979_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533050979_0001_1_00_000000_0
2016-05-18 00:57:31 Starting to run new task attempt: attempt_1463533050979_0001_1_00_000000_0
2016-05-18 00:57:31,184 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533050979_0001_1_00_000000_0
2016-05-18 00:57:31,184 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,184 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:31,184 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79] every/(1/1) ...latform/-1_VALUES_RESULTS, VertexName: AF1520B590E544F8A6EAE66CD734CFE6, VertexParallelism: 1, TaskAttemptID:attempt_1463533050979_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CB9F63A12D0A4459AF8D887F47CFDC56, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=BE08FAEF77DB4358A1A7435A4EF121D5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:31,185 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:31,185 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533050979_0001_1_00_000000_0
2016-05-18 00:57:31,186 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533050979_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:31,186 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533050979_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:31,191 INFO input.MRInput (MRInput.java:initialize(440)) - CB9F63A12D0A4459AF8D887F47CFDC56 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:31,191 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: CB9F63A12D0A4459AF8D887F47CFDC56
2016-05-18 00:57:31,192 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:31,197 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AF1520B590E544F8A6EAE66CD734CFE6, ordinal: 0
2016-05-18 00:57:31,198 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AF1520B590E544F8A6EAE66CD734CFE6, mem on start (mb), free: 310, total: 365, max: 672
2016-05-18 00:57:31,198 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:31,198 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:31,198 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:31,198 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:31,198 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CB9F63A12D0A4459AF8D887F47CFDC56:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [BE08FAEF77DB4358A1A7435A4EF121D5:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:31,198 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:31,199 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:31,199 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:31,199 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CB9F63A12D0A4459AF8D887F47CFDC56:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [BE08FAEF77DB4358A1A7435A4EF121D5:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:31,199 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:31,199 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CB9F63A12D0A4459AF8D887F47CFDC56 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:31,199 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:31,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:31,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CB9F63A12D0A4459AF8D887F47CFDC56
2016-05-18 00:57:31,200 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:31,200 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533050979_0001_1_00_000000_0
2016-05-18 00:57:31,203 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:31,203 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - CB9F63A12D0A4459AF8D887F47CFDC56 initialized RecordReader from event
2016-05-18 00:57:31,204 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:31,204 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: CB9F63A12D0A4459AF8D887F47CFDC56
2016-05-18 00:57:31,204 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: DEC5A2A93BAE4DF99757927DD2066E9B
2016-05-18 00:57:31,204 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] DEC5A2A93BAE4DF99757927DD2066E9B
2016-05-18 00:57:31,205 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:31,208 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:31,208 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - BE08FAEF77DB4358A1A7435A4EF121D5 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@1605605c, comparator=cascading.tuple.hadoop.util.TupleComparator@3af9434f, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:31,208 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for BE08FAEF77DB4358A1A7435A4EF121D5: , UsingHashComparator=false
2016-05-18 00:57:31,220 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:31,220 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:31,221 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for BE08FAEF77DB4358A1A7435A4EF121D5: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:31,221 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - BE08FAEF77DB4358A1A7435A4EF121D5: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:31,221 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:31,221 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AF1520B590E544F8A6EAE66CD734CFE6, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:31,224 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533050979_0001_1_00_000000_0
2016-05-18 00:57:31,224 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AF1520B590E544F8A6EAE66CD734CFE6, mem on close (mb), free: 208, total: 365, max: 672
2016-05-18 00:57:31,224 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - BE08FAEF77DB4358A1A7435A4EF121D5: Starting flush of map output
2016-05-18 00:57:31,225 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - BE08FAEF77DB4358A1A7435A4EF121D5: Span0.length = 20, perItem = 6
2016-05-18 00:57:31,225 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - BE08FAEF77DB4358A1A7435A4EF121D5: done sorting span=0, length=20, time=0
2016-05-18 00:57:31,225 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - BE08FAEF77DB4358A1A7435A4EF121D5: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:31,226 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - BE08FAEF77DB4358A1A7435A4EF121D5: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001_wd/localmode-local-dir/output/attempt_1463533050979_0001_1_00_000000_0_10069_0/file.out
2016-05-18 00:57:31,230 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:31 Completed running task attempt: attempt_1463533050979_0001_1_00_000000_0
2016-05-18 00:57:31,231 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533050979_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:31,231 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533050979_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:31,232 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533050979_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533051222, Process_Duration=2, Process_End_Time=1463533051224, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:31,233 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533050979_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:31,233 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:31,233 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AF1520B590E544F8A6EAE66CD734CFE6, taskAttemptId=attempt_1463533050979_0001_1_00_000000_0, creationTime=1463533051173, allocationTime=1463533051179, startTime=1463533051182, finishTime=1463533051233, timeTaken=51, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:31,233 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533050979_0001_1_00_000000_0
2016-05-18 00:57:31,233 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:31,234 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,234 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533050979_0001_00_000001
2016-05-18 00:57:31,234 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:TASK_FINISHED]: vertexName=AF1520B590E544F8A6EAE66CD734CFE6, taskId=task_1463533050979_0001_1_00_000000, startTime=1463533051182, finishTime=1463533051233, timeTaken=51, status=SUCCEEDED, successfulAttemptID=attempt_1463533050979_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533051222, Process_Duration=2, Process_End_Time=1463533051224, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:31,235 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:31,235 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6]
2016-05-18 00:57:31,236 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:VERTEX_FINISHED]: vertexName=AF1520B590E544F8A6EAE66CD734CFE6, vertexId=vertex_1463533050979_0001_1_00, initRequestedTime=1463533051154, initedTime=1463533051167, startRequestedTime=1463533051156, startedTime=1463533051169, finishTime=1463533051235, timeTaken=66, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533051222, Process_Duration=2, Process_End_Time=1463533051224, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533051182, firstTasksToStart=[ task_1463533050979_0001_1_00_000000 ], lastTaskFinishTime=1463533051233, lastTasksToFinish=[ task_1463533050979_0001_1_00_000000 ], minTaskDuration=51, maxTaskDuration=51, avgTaskDuration=51.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533050979_0001_1_00_000000 ], longestDurationTasks=[ task_1463533050979_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:31,236 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:31,237 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533050979_0001_00_000001, stoppedTime=1463533051237, exitStatus=0
2016-05-18 00:57:31,237 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] attempt: attempt_1463533050979_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:31,237 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533050979_0001_1_00 [AF1520B590E544F8A6EAE66CD734CFE6] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:31,237 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: BE08FAEF77DB4358A1A7435A4EF121D5
2016-05-18 00:57:31,237 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:31,237 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=AF1520B590E544F8A6EAE66CD734CFE6, destinationVertexName=BE08FAEF77DB4358A1A7435A4EF121D5. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:31,237 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533050979_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:31,237 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,238 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:TASK_STARTED]: vertexName=BE08FAEF77DB4358A1A7435A4EF121D5, taskId=task_1463533050979_0001_1_01_000000, scheduledTime=1463533051238, launchTime=1463533051238
2016-05-18 00:57:31,238 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,240 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533050979_0001_00_000002, launchTime=1463533051240
2016-05-18 00:57:31,241 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533050979_0001_00_000002
2016-05-18 00:57:31,242 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533050979_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533050979_0001_1_01_000000_0
2016-05-18 00:57:31 Starting to run new task attempt: attempt_1463533050979_0001_1_01_000000_0
2016-05-18 00:57:31,242 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533050979_0001_1_01_000000_0
2016-05-18 00:57:31,242 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,242 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:31,243 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79] every/(1/1) ...latform/-1_VALUES_RESULTS, VertexName: BE08FAEF77DB4358A1A7435A4EF121D5, VertexParallelism: 1, TaskAttemptID:attempt_1463533050979_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AF1520B590E544F8A6EAE66CD734CFE6, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=63ED769EF57F49A388509B50421F2202, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:31,243 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:31,243 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533050979_0001_1_01_000000_0] started. Is using containerId: [container_1463533050979_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:31,244 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BE08FAEF77DB4358A1A7435A4EF121D5, taskAttemptId=attempt_1463533050979_0001_1_01_000000_0, startTime=1463533051242, containerId=container_1463533050979_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:31,245 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533050979_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:31,245 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533050979_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:31,245 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533050979_0001_00_000001, Exiting
2016-05-18 00:57:31,245 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533050979_0001_00_000001
2016-05-18 00:57:31,245 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533050979_0001_00_000001
2016-05-18 00:57:31,248 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533050979_0001_1_01_000000_0
2016-05-18 00:57:31,249 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533050979_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,249 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533050979_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:31,249 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533050979_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:31,250 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533050979_0001_00_000001 completed successfully
2016-05-18 00:57:31,251 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:31,252 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:31,259 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BE08FAEF77DB4358A1A7435A4EF121D5, ordinal: 1
2016-05-18 00:57:31,259 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BE08FAEF77DB4358A1A7435A4EF121D5, mem on start (mb), free: 203, total: 365, max: 672
2016-05-18 00:57:31,260 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:31,260 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:31,260 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:31,263 INFO output.MROutput (MROutput.java:initialize(421)) - 63ED769EF57F49A388509B50421F2202: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:31,263 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:31,263 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[AF1520B590E544F8A6EAE66CD734CFE6:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [63ED769EF57F49A388509B50421F2202:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:31,264 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:31,264 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:31,264 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:31,264 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[AF1520B590E544F8A6EAE66CD734CFE6:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [63ED769EF57F49A388509B50421F2202:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:31,264 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:31,264 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AF1520B590E544F8A6EAE66CD734CFE6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:31,265 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:31,265 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:31,265 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - AF1520B590E544F8A6EAE66CD734CFE6: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:31,265 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - AF1520B590E544F8A6EAE66CD734CFE6: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:31,266 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: AF1520B590E544F8A6EAE66CD734CFE6 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:31,267 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AF1520B590E544F8A6EAE66CD734CFE6
2016-05-18 00:57:31,269 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:31,269 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533050979_0001_1_01_000000_0
2016-05-18 00:57:31,271 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AF1520B590E544F8A6EAE66CD734CFE6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:31,272 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:31,272 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: DEC5A2A93BAE4DF99757927DD2066E9B
2016-05-18 00:57:31,272 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533050979_0001_1_00_000000_0_10069} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533051272, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:31,272 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_RESULTS"], id: 63ED769EF57F49A388509B50421F2202
2016-05-18 00:57:31,273 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_RESULTS"]
2016-05-18 00:57:31,273 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] DEC5A2A93BAE4DF99757927DD2066E9B, for 1 inputs
2016-05-18 00:57:31,273 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : AF1520B590E544F8A6EAE66CD734CFE6
2016-05-18 00:57:31,274 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:31,274 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: AF1520B590E544F8A6EAE66CD734CFE6, wasInterrupted=false
2016-05-18 00:57:31,275 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:31,276 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:31,277 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:31,277 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001_wd/localmode-local-dir/output/attempt_1463533050979_0001_1_00_000000_0_10069/file.out, len=181, isLocal=true
2016-05-18 00:57:31,277 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:31,278 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:31,278 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:31,278 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:31,279 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : AF1520B590E544F8A6EAE66CD734CFE6
2016-05-18 00:57:31,279 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BE08FAEF77DB4358A1A7435A4EF121D5, all 1 inputs ready in: 00:00:00.006
2016-05-18 00:57:31,279 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - AF1520B590E544F8A6EAE66CD734CFE6: Shuffle Runner thread complete
2016-05-18 00:57:31,280 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:31,280 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - AF1520B590E544F8A6EAE66CD734CFE6: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:31,282 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:31,283 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:31,283 INFO output.MROutput (MROutput.java:close(526)) - 63ED769EF57F49A388509B50421F2202 closed
2016-05-18 00:57:31,284 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533050979_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:31,285 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533050979_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:31,285 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533050979_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533051279, Process_Duration=6, Process_End_Time=1463533051285, Read_Duration=1, Tuples_Read=1]]
2016-05-18 00:57:31,285 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:31,285 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BE08FAEF77DB4358A1A7435A4EF121D5, mem on close (mb), free: 199, total: 365, max: 672
2016-05-18 00:57:31,285 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=BE08FAEF77DB4358A1A7435A4EF121D5, index=1, interruptedStatus=false
2016-05-18 00:57:31,286 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: AF1520B590E544F8A6EAE66CD734CFE6
2016-05-18 00:57:31,286 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AF1520B590E544F8A6EAE66CD734CFE6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:31,286 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=BE08FAEF77DB4358A1A7435A4EF121D5, sourceVertex=AF1520B590E544F8A6EAE66CD734CFE6, interruptedStatus=true
2016-05-18 00:57:31,286 INFO output.MROutput (MROutput.java:close(526)) - 63ED769EF57F49A388509B50421F2202 closed
2016-05-18 00:57:31,286 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=BE08FAEF77DB4358A1A7435A4EF121D5, sourceVertex=63ED769EF57F49A388509B50421F2202, interruptedStatus=true
2016-05-18 00:57:31,288 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533050979_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:31,288 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:31,288 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533050979_0001_00_000002
2016-05-18 00:57:31,288 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BE08FAEF77DB4358A1A7435A4EF121D5, taskAttemptId=attempt_1463533050979_0001_1_01_000000_0, creationTime=1463533051238, allocationTime=1463533051239, startTime=1463533051242, finishTime=1463533051288, timeTaken=46, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533050979_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533051279, Process_Duration=6, Process_End_Time=1463533051285, Read_Duration=1, Tuples_Read=1
2016-05-18 00:57:31,288 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533050979_0001_00_000002
2016-05-18 00:57:31,289 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533050979_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:31,289 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:TASK_FINISHED]: vertexName=BE08FAEF77DB4358A1A7435A4EF121D5, taskId=task_1463533050979_0001_1_01_000000, startTime=1463533051242, finishTime=1463533051289, timeTaken=47, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533050979_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:31,289 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] because task failed: task_1463533050979_0001_1_01_000000
2016-05-18 00:57:31,289 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:31,290 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:31,290 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:31,290 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,290 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=63ED769EF57F49A388509B50421F2202, vertexId=vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,290 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533050979_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:31,291 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533050979_0001_00_000002 completed but with errors
2016-05-18 00:57:31,293 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:VERTEX_FINISHED]: vertexName=BE08FAEF77DB4358A1A7435A4EF121D5, vertexId=vertex_1463533050979_0001_1_01, initRequestedTime=1463533051156, initedTime=1463533051160, startRequestedTime=1463533051170, startedTime=1463533051171, finishTime=1463533051292, timeTaken=121, status=FAILED, diagnostics=Task failed, taskId=task_1463533050979_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533050979_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533051242, firstTasksToStart=[ task_1463533050979_0001_1_01_000000 ], lastTaskFinishTime=1463533051288, lastTasksToFinish=[ task_1463533050979_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:31,294 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:31,295 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533050979_0001_00_000002, stoppedTime=1463533051295, exitStatus=-1000
2016-05-18 00:57:31,295 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533050979_0001_1_01_000000_0 failed: true container: container_1463533050979_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:31,295 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533050979_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:31,296 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:31,296 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:31,296 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:31,298 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533050979_0001_1, queueSize=0
2016-05-18 00:57:31,299 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533050979_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533050979_0001_1, startTime=1463533051154, finishTime=1463533051297, timeTaken=143, status=FAILED, diagnostics=Vertex failed, vertexName=BE08FAEF77DB4358A1A7435A4EF121D5, vertexId=vertex_1463533050979_0001_1_01, diagnostics=[Task failed, taskId=task_1463533050979_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533050979_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=240, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533051222, Process_Duration=2, Process_End_Time=1463533051224, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:31,299 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533050979_0001_1_01 [BE08FAEF77DB4358A1A7435A4EF121D5]
2016-05-18 00:57:31,299 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533050979_0001_1 finished with state: FAILED
2016-05-18 00:57:31,300 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533050979_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:31,301 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533050979_0001_00_000002
2016-05-18 00:57:31 Completed Dag: dag_1463533050979_0001_1
2016-05-18 00:57:31,301 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533050979_0001_1, dagState=FAILED
2016-05-18 00:57:31,302 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533050979_0001_1_post
2016-05-18 00:57:31,302 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,302 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:31,302 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:31,302 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79] every/(1/1) ...latform/-1_VALUES_RESULTS, with id=dag_1463533050979_0001_1
2016-05-18 00:57:31,302 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79] every/(1/1) ...latform/-1_VALUES_RESULTS, with id=dag_1463533050979_0001_1
2016-05-18 00:57:31,311 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...latform/-1_VALUES_RESULTS, applicationId=application_1463533050979_0001
2016-05-18 00:57:31,313 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:31,313 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:31,313 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:31,314 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:31,314 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:31,314 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:31,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:31,315 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:31,315 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:31,315 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:31,315 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:31,316 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:31,316 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:31,316 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:31,320 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/70C0150245894C209178FCF2C18FFD68/BAE3F9ABA67E4C3E9E0524BA0F4B4A79/.tez/application_1463533050979_0001
2016-05-18 00:57:31,320 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:31,320 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...latform/-1_VALUES_RESULTS
2016-05-18 00:57:31,321 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:31,321 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_RESULTS/_temporary
2016-05-18 00:57:31,321 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.360
2016-05-18 00:57:31,321 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_VALUES_RESULTS
2016-05-18 00:57:31,359 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:31,359 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:31,414 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 00:57:31,425 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.026
2016-05-18 00:57:31,425 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:31,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:31,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:31,449 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:31,458 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_ALL
2016-05-18 00:57:31,532 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:31,533 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:31,533 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_ALL"]
2016-05-18 00:57:31,533 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:31,534 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:31,534 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:31,536 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...blyplatform/-1_VALUES_ALL
2016-05-18 00:57:31,536 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:31,550 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:31,554 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:31,555 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:31,555 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:31,564 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001 doesn't exist and is created
2016-05-18 00:57:31,579 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:31,580 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:31,580 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001_wd
2016-05-18 00:57:31,591 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533051555_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:31,592 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:31,592 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:31,593 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:31,594 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:31,594 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:31,594 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:31,595 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:31,596 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:31,597 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:31,597 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:31,597 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:31,597 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:31,597 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:31,598 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:31,598 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:31,598 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:31,598 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:31,599 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:31,599 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:31,600 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:31,601 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:31,602 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:31,602 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001_wd/localmode-log-dir
2016-05-18 00:57:31,602 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001_wd/localmode-log-dir/history.txt.appattempt_1463533051555_0001_000000, maxErrors=10
2016-05-18 00:57:31,603 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:31,603 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533051555_0001_000000, appSubmitTime=1463533051591, launchTime=1463533051591
2016-05-18 00:57:31,607 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:31,613 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:39375
2016-05-18 00:57:31,613 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533051555
2016-05-18 00:57:31,614 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533051555_0001_000000, startTime=1463533051614
2016-05-18 00:57:31,614 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:31,680 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:31,680 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:31,680 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...blyplatform/-1_VALUES_ALL, applicationId=application_1463533051555_0001, dagName=[EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3] every/(1/1) ...blyplatform/-1_VALUES_ALL
2016-05-18 00:57:31,685 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:31,685 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:31,697 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3] every/(1/1) ...blyplatform/-1_VALUES_ALL
2016-05-18 00:57:31,697 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:31,698 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533051555_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001_wd/localmode-log-dir/dag_1463533051555_0001_1.dot
2016-05-18 00:57:31,698 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001_wd/localmode-log-dir/dag_1463533051555_0001_1-tez-dag.pb.txt
2016-05-18 00:57:31,705 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533051555_0001_1
2016-05-18 00:57:31,705 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,705 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3] every/(1/1) ...blyplatform/-1_VALUES_ALL
2016-05-18 00:57:31 Running Dag: dag_1463533051555_0001_1
2016-05-18 00:57:31,714 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533051555_0001_1, submitTime=1463533051697
2016-05-18 00:57:31,716 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001/tez-conf.pb" } size: 36913 timestamp: 1463533051000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:31,717 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,717 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:31,718 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001/tez-conf.pb" } size: 36913 timestamp: 1463533051000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:31,718 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,719 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:31,719 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:31,720 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533051555_0001_1, initTime=1463533051720
2016-05-18 00:57:31,720 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533051555_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:31,721 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:31,721 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:DAG_STARTED]: dagID=dag_1463533051555_0001_1, startTime=1463533051721
2016-05-18 00:57:31,721 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533051555_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:31,721 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...blyplatform/-1_VALUES_ALL, applicationId=application_1463533051555_0001, dagName=[EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3] every/(1/1) ...blyplatform/-1_VALUES_ALL
2016-05-18 00:57:31,721 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: BCD1AF6F61CC40C7A4D9FA35BFDACB29 : {29DDAC17FEC9467AAD20AF1EB4B5E06A={InputName=29DDAC17FEC9467AAD20AF1EB4B5E06A}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:31,722 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 29DDAC17FEC9467AAD20AF1EB4B5E06A, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:31,722 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,722 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,722 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,723 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,723 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:31,723 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 29DDAC17FEC9467AAD20AF1EB4B5E06A on vertex vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,723 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,726 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 29DDAC17FEC9467AAD20AF1EB4B5E06A asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:31,727 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:31,727 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,727 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,728 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:31,728 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:31,729 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:31,729 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:31,730 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 29DDAC17FEC9467AAD20AF1EB4B5E06A on vertex vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,733 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533051555_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:31,734 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA], numAdditionalOutputs=1
2016-05-18 00:57:31,734 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=16B2A93A83464941A88282FDFE802505, vertex=vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:31,736 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F18C76E21F8844C5BF6BA4500509EADA:16B2A93A83464941A88282FDFE802505 using oldmapred API
2016-05-18 00:57:31,738 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=16B2A93A83464941A88282FDFE802505, vertexName=F18C76E21F8844C5BF6BA4500509EADA, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:31,740 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F18C76E21F8844C5BF6BA4500509EADA, vertexId=vertex_1463533051555_0001_1_01, initRequestedTime=1463533051723, initedTime=1463533051734, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:31,741 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:31,743 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:31,744 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {29DDAC17FEC9467AAD20AF1EB4B5E06A=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:31,745 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] parallelism set to 1
2016-05-18 00:57:31,746 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:31,745 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533051555_0001, with dag id: dag_1463533051555_0001_1
2016-05-18 00:57:31,747 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BCD1AF6F61CC40C7A4D9FA35BFDACB29, vertexId=vertex_1463533051555_0001_1_00, initRequestedTime=1463533051721, initedTime=1463533051747, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:31,747 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] with distanceFromRoot: 0
2016-05-18 00:57:31,747 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:31,748 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533051555_0001_1_00, startRequestedTime=1463533051723, startedTime=1463533051748
2016-05-18 00:57:31,762 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533051555_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:31,763 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:31,763 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533051555_0001_1_00 for vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] numStartedSources: 1 numSources: 1
2016-05-18 00:57:31,763 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] with distanceFromRoot: 1
2016-05-18 00:57:31,763 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533051555_0001_1_01, startRequestedTime=1463533051763, startedTime=1463533051763
2016-05-18 00:57:31,764 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:31,765 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F18C76E21F8844C5BF6BA4500509EADA with 0 source tasks and 1 pending tasks
2016-05-18 00:57:31,765 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in BCD1AF6F61CC40C7A4D9FA35BFDACB29
2016-05-18 00:57:31,765 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,766 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:TASK_STARTED]: vertexName=BCD1AF6F61CC40C7A4D9FA35BFDACB29, taskId=task_1463533051555_0001_1_00_000000, scheduledTime=1463533051766, launchTime=1463533051766
2016-05-18 00:57:31,766 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,768 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BCD1AF6F61CC40C7A4D9FA35BFDACB29 in vertex: F18C76E21F8844C5BF6BA4500509EADA numBipartiteSourceTasks: 1
2016-05-18 00:57:31,769 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:31,771 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533051555_0001_00_000001, launchTime=1463533051771
2016-05-18 00:57:31,773 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533051555_0001_00_000001
2016-05-18 00:57:31,773 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533051555_0001_1_00_000000_0] started. Is using containerId: [container_1463533051555_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:31,774 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BCD1AF6F61CC40C7A4D9FA35BFDACB29, taskAttemptId=attempt_1463533051555_0001_1_00_000000_0, startTime=1463533051773, containerId=container_1463533051555_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:31,774 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533051555_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533051555_0001_1_00_000000_0
2016-05-18 00:57:31 Starting to run new task attempt: attempt_1463533051555_0001_1_00_000000_0
2016-05-18 00:57:31,774 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533051555_0001_1_00_000000_0
2016-05-18 00:57:31,775 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,775 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:31,775 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3] every/(1/1) ...blyplatform/-1_VALUES_ALL, VertexName: BCD1AF6F61CC40C7A4D9FA35BFDACB29, VertexParallelism: 1, TaskAttemptID:attempt_1463533051555_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=29DDAC17FEC9467AAD20AF1EB4B5E06A, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F18C76E21F8844C5BF6BA4500509EADA, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:31,775 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:31,777 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533051555_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:31,777 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533051555_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:31,778 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533051555_0001_1_00_000000_0
2016-05-18 00:57:31,783 INFO input.MRInput (MRInput.java:initialize(440)) - 29DDAC17FEC9467AAD20AF1EB4B5E06A using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:31,783 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 29DDAC17FEC9467AAD20AF1EB4B5E06A
2016-05-18 00:57:31,787 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:31,797 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BCD1AF6F61CC40C7A4D9FA35BFDACB29, ordinal: 0
2016-05-18 00:57:31,798 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BCD1AF6F61CC40C7A4D9FA35BFDACB29, mem on start (mb), free: 324, total: 365, max: 672
2016-05-18 00:57:31,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:31,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:31,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:31,799 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:31,799 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[29DDAC17FEC9467AAD20AF1EB4B5E06A:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F18C76E21F8844C5BF6BA4500509EADA:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:31,799 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:31,799 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:31,801 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:31,801 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[29DDAC17FEC9467AAD20AF1EB4B5E06A:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F18C76E21F8844C5BF6BA4500509EADA:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:31,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:31,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 29DDAC17FEC9467AAD20AF1EB4B5E06A
2016-05-18 00:57:31,801 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 29DDAC17FEC9467AAD20AF1EB4B5E06A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:31,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:31,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:31,804 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:31,804 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533051555_0001_1_00_000000_0
2016-05-18 00:57:31,807 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:31,808 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 29DDAC17FEC9467AAD20AF1EB4B5E06A
2016-05-18 00:57:31,808 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: DA27C8C495BD4D2494D4DE4D6A6816D0
2016-05-18 00:57:31,808 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] DA27C8C495BD4D2494D4DE4D6A6816D0
2016-05-18 00:57:31,808 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:31,810 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 29DDAC17FEC9467AAD20AF1EB4B5E06A initialized RecordReader from event
2016-05-18 00:57:31,809 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:31,815 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:31,815 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F18C76E21F8844C5BF6BA4500509EADA using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@5d22005a, comparator=cascading.tuple.hadoop.util.TupleComparator@10c3e8ee, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:31,816 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F18C76E21F8844C5BF6BA4500509EADA: , UsingHashComparator=false
2016-05-18 00:57:31,831 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:31,831 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:31,832 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F18C76E21F8844C5BF6BA4500509EADA: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:31,832 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F18C76E21F8844C5BF6BA4500509EADA: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:31,832 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:31,835 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BCD1AF6F61CC40C7A4D9FA35BFDACB29, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:31,839 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533051555_0001_1_00_000000_0
2016-05-18 00:57:31,839 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BCD1AF6F61CC40C7A4D9FA35BFDACB29, mem on close (mb), free: 222, total: 365, max: 672
2016-05-18 00:57:31,840 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F18C76E21F8844C5BF6BA4500509EADA: Starting flush of map output
2016-05-18 00:57:31,840 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F18C76E21F8844C5BF6BA4500509EADA: Span0.length = 20, perItem = 6
2016-05-18 00:57:31,841 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F18C76E21F8844C5BF6BA4500509EADA: done sorting span=0, length=20, time=0
2016-05-18 00:57:31,842 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F18C76E21F8844C5BF6BA4500509EADA: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:31,844 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F18C76E21F8844C5BF6BA4500509EADA: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001_wd/localmode-local-dir/output/attempt_1463533051555_0001_1_00_000000_0_10075_0/file.out
2016-05-18 00:57:31,850 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:31 Completed running task attempt: attempt_1463533051555_0001_1_00_000000_0
2016-05-18 00:57:31,852 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533051555_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:31,852 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533051555_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:31,853 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533051555_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533051835, Process_Duration=4, Process_End_Time=1463533051839, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=3][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:31,854 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533051555_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:31,854 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BCD1AF6F61CC40C7A4D9FA35BFDACB29, taskAttemptId=attempt_1463533051555_0001_1_00_000000_0, creationTime=1463533051766, allocationTime=1463533051770, startTime=1463533051773, finishTime=1463533051854, timeTaken=81, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:31,854 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:31,855 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:31,855 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,855 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533051555_0001_1_00_000000_0
2016-05-18 00:57:31,856 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:TASK_FINISHED]: vertexName=BCD1AF6F61CC40C7A4D9FA35BFDACB29, taskId=task_1463533051555_0001_1_00_000000, startTime=1463533051773, finishTime=1463533051855, timeTaken=82, status=SUCCEEDED, successfulAttemptID=attempt_1463533051555_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533051835, Process_Duration=4, Process_End_Time=1463533051839, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:31,856 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:31,856 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29]
2016-05-18 00:57:31,858 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:VERTEX_FINISHED]: vertexName=BCD1AF6F61CC40C7A4D9FA35BFDACB29, vertexId=vertex_1463533051555_0001_1_00, initRequestedTime=1463533051721, initedTime=1463533051747, startRequestedTime=1463533051723, startedTime=1463533051748, finishTime=1463533051856, timeTaken=108, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533051835, Process_Duration=4, Process_End_Time=1463533051839, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533051773, firstTasksToStart=[ task_1463533051555_0001_1_00_000000 ], lastTaskFinishTime=1463533051854, lastTasksToFinish=[ task_1463533051555_0001_1_00_000000 ], minTaskDuration=81, maxTaskDuration=81, avgTaskDuration=81.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533051555_0001_1_00_000000 ], longestDurationTasks=[ task_1463533051555_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:31,859 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:31,859 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] attempt: attempt_1463533051555_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:31,859 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533051555_0001_00_000001, stoppedTime=1463533051859, exitStatus=0
2016-05-18 00:57:31,859 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533051555_0001_1_00 [BCD1AF6F61CC40C7A4D9FA35BFDACB29] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:31,859 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:31,860 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F18C76E21F8844C5BF6BA4500509EADA
2016-05-18 00:57:31,860 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533051555_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:31,859 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533051555_0001_00_000001
2016-05-18 00:57:31,860 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BCD1AF6F61CC40C7A4D9FA35BFDACB29, destinationVertexName=F18C76E21F8844C5BF6BA4500509EADA. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:31,860 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,861 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:TASK_STARTED]: vertexName=F18C76E21F8844C5BF6BA4500509EADA, taskId=task_1463533051555_0001_1_01_000000, scheduledTime=1463533051861, launchTime=1463533051861
2016-05-18 00:57:31,861 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,861 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533051555_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:31,862 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533051555_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:31,862 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533051555_0001_00_000001, Exiting
2016-05-18 00:57:31,862 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533051555_0001_00_000001
2016-05-18 00:57:31,862 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533051555_0001_00_000001
2016-05-18 00:57:31,864 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533051555_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:31,864 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533051555_0001_00_000001 completed successfully
2016-05-18 00:57:31,865 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533051555_0001_00_000002, launchTime=1463533051865
2016-05-18 00:57:31,866 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533051555_0001_00_000002
2016-05-18 00:57:31,867 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533051555_0001_1_01_000000_0] started. Is using containerId: [container_1463533051555_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:31,867 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F18C76E21F8844C5BF6BA4500509EADA, taskAttemptId=attempt_1463533051555_0001_1_01_000000_0, startTime=1463533051867, containerId=container_1463533051555_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:31,868 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533051555_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533051555_0001_1_01_000000_0
2016-05-18 00:57:31 Starting to run new task attempt: attempt_1463533051555_0001_1_01_000000_0
2016-05-18 00:57:31,869 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533051555_0001_1_01_000000_0
2016-05-18 00:57:31,869 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,869 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:31,869 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3] every/(1/1) ...blyplatform/-1_VALUES_ALL, VertexName: F18C76E21F8844C5BF6BA4500509EADA, VertexParallelism: 1, TaskAttemptID:attempt_1463533051555_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BCD1AF6F61CC40C7A4D9FA35BFDACB29, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=16B2A93A83464941A88282FDFE802505, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:31,870 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:31,870 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533051555_0001_1_01_000000_0
2016-05-18 00:57:31,871 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533051555_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,872 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533051555_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:31,875 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:31,878 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:31,888 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F18C76E21F8844C5BF6BA4500509EADA, ordinal: 1
2016-05-18 00:57:31,888 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F18C76E21F8844C5BF6BA4500509EADA, mem on start (mb), free: 217, total: 365, max: 672
2016-05-18 00:57:31,888 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:31,889 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:31,889 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:31,888 INFO output.MROutput (MROutput.java:initialize(421)) - 16B2A93A83464941A88282FDFE802505: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:31,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:31,890 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BCD1AF6F61CC40C7A4D9FA35BFDACB29:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [16B2A93A83464941A88282FDFE802505:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:31,891 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:31,891 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:31,891 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:31,892 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BCD1AF6F61CC40C7A4D9FA35BFDACB29:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [16B2A93A83464941A88282FDFE802505:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:31,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:31,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BCD1AF6F61CC40C7A4D9FA35BFDACB29 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:31,893 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:31,893 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:31,894 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BCD1AF6F61CC40C7A4D9FA35BFDACB29: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:31,894 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BCD1AF6F61CC40C7A4D9FA35BFDACB29: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:31,895 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BCD1AF6F61CC40C7A4D9FA35BFDACB29 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:31,896 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BCD1AF6F61CC40C7A4D9FA35BFDACB29
2016-05-18 00:57:31,896 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:31,896 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533051555_0001_1_01_000000_0
2016-05-18 00:57:31,902 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BCD1AF6F61CC40C7A4D9FA35BFDACB29: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:31,902 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:31,903 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: DA27C8C495BD4D2494D4DE4D6A6816D0
2016-05-18 00:57:31,903 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_ALL"], id: 16B2A93A83464941A88282FDFE802505
2016-05-18 00:57:31,904 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_ALL"]
2016-05-18 00:57:31,904 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] DA27C8C495BD4D2494D4DE4D6A6816D0, for 1 inputs
2016-05-18 00:57:31,905 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533051555_0001_1_00_000000_0_10075} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533051905, TimeTaken=2, Rate=0.09 MB/s
2016-05-18 00:57:31,906 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BCD1AF6F61CC40C7A4D9FA35BFDACB29
2016-05-18 00:57:31,906 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:31,907 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BCD1AF6F61CC40C7A4D9FA35BFDACB29, wasInterrupted=false
2016-05-18 00:57:31,907 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:31,908 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:31,909 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:31,910 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001_wd/localmode-local-dir/output/attempt_1463533051555_0001_1_00_000000_0_10075/file.out, len=181, isLocal=true
2016-05-18 00:57:31,910 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:31,910 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:31,910 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:31,910 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:31,911 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BCD1AF6F61CC40C7A4D9FA35BFDACB29
2016-05-18 00:57:31,911 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BCD1AF6F61CC40C7A4D9FA35BFDACB29: Shuffle Runner thread complete
2016-05-18 00:57:31,911 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F18C76E21F8844C5BF6BA4500509EADA, all 1 inputs ready in: 00:00:00.006
2016-05-18 00:57:31,915 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:31,915 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BCD1AF6F61CC40C7A4D9FA35BFDACB29: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:31,918 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:31,921 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:31,926 INFO output.MROutput (MROutput.java:close(526)) - 16B2A93A83464941A88282FDFE802505 closed
2016-05-18 00:57:31,927 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533051555_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:31,927 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533051555_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:31,928 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533051555_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533051914, Process_Duration=13, Process_End_Time=1463533051927, Read_Duration=1, Tuples_Read=1]]
2016-05-18 00:57:31,928 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:31,929 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F18C76E21F8844C5BF6BA4500509EADA, mem on close (mb), free: 213, total: 365, max: 672
2016-05-18 00:57:31,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=F18C76E21F8844C5BF6BA4500509EADA, index=1, interruptedStatus=false
2016-05-18 00:57:31,930 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BCD1AF6F61CC40C7A4D9FA35BFDACB29
2016-05-18 00:57:31,930 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BCD1AF6F61CC40C7A4D9FA35BFDACB29: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:31,930 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=F18C76E21F8844C5BF6BA4500509EADA, sourceVertex=BCD1AF6F61CC40C7A4D9FA35BFDACB29, interruptedStatus=true
2016-05-18 00:57:31,930 INFO output.MROutput (MROutput.java:close(526)) - 16B2A93A83464941A88282FDFE802505 closed
2016-05-18 00:57:31,931 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=F18C76E21F8844C5BF6BA4500509EADA, sourceVertex=16B2A93A83464941A88282FDFE802505, interruptedStatus=true
2016-05-18 00:57:31,932 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533051555_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:31,932 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:31,932 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533051555_0001_00_000002
2016-05-18 00:57:31,932 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533051555_0001_00_000002
2016-05-18 00:57:31,933 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F18C76E21F8844C5BF6BA4500509EADA, taskAttemptId=attempt_1463533051555_0001_1_01_000000_0, creationTime=1463533051861, allocationTime=1463533051862, startTime=1463533051867, finishTime=1463533051932, timeTaken=65, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533051555_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533051914, Process_Duration=13, Process_End_Time=1463533051927, Read_Duration=1, Tuples_Read=1
2016-05-18 00:57:31,933 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533051555_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:31,933 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:TASK_FINISHED]: vertexName=F18C76E21F8844C5BF6BA4500509EADA, taskId=task_1463533051555_0001_1_01_000000, startTime=1463533051867, finishTime=1463533051933, timeTaken=66, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533051555_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:31,933 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] because task failed: task_1463533051555_0001_1_01_000000
2016-05-18 00:57:31,933 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:31,933 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:31,934 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:31,934 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,934 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=16B2A93A83464941A88282FDFE802505, vertexId=vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,934 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533051555_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:31,934 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533051555_0001_00_000002 completed but with errors
2016-05-18 00:57:31,935 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:VERTEX_FINISHED]: vertexName=F18C76E21F8844C5BF6BA4500509EADA, vertexId=vertex_1463533051555_0001_1_01, initRequestedTime=1463533051723, initedTime=1463533051734, startRequestedTime=1463533051763, startedTime=1463533051763, finishTime=1463533051935, timeTaken=172, status=FAILED, diagnostics=Task failed, taskId=task_1463533051555_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533051555_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533051867, firstTasksToStart=[ task_1463533051555_0001_1_01_000000 ], lastTaskFinishTime=1463533051932, lastTasksToFinish=[ task_1463533051555_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:31,935 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:31,935 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533051555_0001_00_000002, stoppedTime=1463533051935, exitStatus=-1000
2016-05-18 00:57:31,935 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533051555_0001_1_01_000000_0 failed: true container: container_1463533051555_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:31,935 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533051555_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:31,936 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:31,936 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:31,936 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:31,939 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533051555_0001_1, queueSize=0
2016-05-18 00:57:31,939 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533051555_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533051555_0001_1, startTime=1463533051721, finishTime=1463533051938, timeTaken=217, status=FAILED, diagnostics=Vertex failed, vertexName=F18C76E21F8844C5BF6BA4500509EADA, vertexId=vertex_1463533051555_0001_1_01, diagnostics=[Task failed, taskId=task_1463533051555_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533051555_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=570, AM_GC_TIME_MILLIS=12, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533051835, Process_Duration=4, Process_End_Time=1463533051839, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:31,940 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533051555_0001_1_01 [F18C76E21F8844C5BF6BA4500509EADA]
2016-05-18 00:57:31,940 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533051555_0001_1 finished with state: FAILED
2016-05-18 00:57:31,940 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533051555_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:31,943 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533051555_0001_00_000002
2016-05-18 00:57:31 Completed Dag: dag_1463533051555_0001_1
2016-05-18 00:57:31,944 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533051555_0001_1, dagState=FAILED
2016-05-18 00:57:31,944 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533051555_0001_1_post
2016-05-18 00:57:31,944 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:31,944 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:31,944 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:31,944 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3] every/(1/1) ...blyplatform/-1_VALUES_ALL, with id=dag_1463533051555_0001_1
2016-05-18 00:57:31,945 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3] every/(1/1) ...blyplatform/-1_VALUES_ALL, with id=dag_1463533051555_0001_1
2016-05-18 00:57:31,948 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...blyplatform/-1_VALUES_ALL, applicationId=application_1463533051555_0001
2016-05-18 00:57:31,951 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:31,951 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:31,951 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:31,951 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:31,951 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:31,952 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:31,953 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:31,953 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:31,954 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:31,954 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:31,954 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:31,954 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:31,955 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:31,956 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:31,960 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/EE1BAEA95A8B499B80DD99F75F3AA91A/B874BF36ED994CD7B7032E44C9E4CBE3/.tez/application_1463533051555_0001
2016-05-18 00:57:31,961 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:31,962 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...blyplatform/-1_VALUES_ALL
2016-05-18 00:57:31,963 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:31,963 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_ALL/_temporary
2016-05-18 00:57:31,963 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.431
2016-05-18 00:57:31,964 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_VALUES_ALL
2016-05-18 00:57:31,989 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:31,989 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:32,028 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:32,031 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 00:57:32,031 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:32,032 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:32,032 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:32,048 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:32,055 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_REPLACE
2016-05-18 00:57:32,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:32,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:32,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_REPLACE"]
2016-05-18 00:57:32,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:32,108 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:32,108 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:32,109 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...latform/-1_VALUES_REPLACE
2016-05-18 00:57:32,109 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:32,122 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:32,126 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:32,126 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:32,127 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:32,135 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001 doesn't exist and is created
2016-05-18 00:57:32,150 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:32,150 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:32,151 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001_wd
2016-05-18 00:57:32,161 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533052126_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:32,161 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:32,161 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:32,163 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:32,163 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:32,163 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:32,163 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:32,165 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:32,166 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:32,166 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:32,166 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:32,167 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:32,167 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:32,167 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:32,167 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:32,168 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:32,168 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:32,169 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:32,169 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:32,169 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:32,170 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:32,171 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:32,171 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:32,171 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001_wd/localmode-log-dir
2016-05-18 00:57:32,172 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001_wd/localmode-log-dir/history.txt.appattempt_1463533052126_0001_000000, maxErrors=10
2016-05-18 00:57:32,172 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:32,172 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533052126_0001_000000, appSubmitTime=1463533052160, launchTime=1463533052160
2016-05-18 00:57:32,175 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:32,179 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:34329
2016-05-18 00:57:32,179 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533052126
2016-05-18 00:57:32,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533052126_0001_000000, startTime=1463533052180
2016-05-18 00:57:32,181 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:32,250 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:32,250 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:32,251 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...latform/-1_VALUES_REPLACE, applicationId=application_1463533052126_0001, dagName=[D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700] every/(1/1) ...latform/-1_VALUES_REPLACE
2016-05-18 00:57:32,267 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700] every/(1/1) ...latform/-1_VALUES_REPLACE
2016-05-18 00:57:32,267 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:32,268 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533052126_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001_wd/localmode-log-dir/dag_1463533052126_0001_1.dot
2016-05-18 00:57:32,269 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001_wd/localmode-log-dir/dag_1463533052126_0001_1-tez-dag.pb.txt
2016-05-18 00:57:32,275 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533052126_0001_1
2016-05-18 00:57:32,275 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,275 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700] every/(1/1) ...latform/-1_VALUES_REPLACE
2016-05-18 00:57:32 Running Dag: dag_1463533052126_0001_1
2016-05-18 00:57:32,284 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533052126_0001_1, submitTime=1463533052267
2016-05-18 00:57:32,285 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001/tez-conf.pb" } size: 36913 timestamp: 1463533052000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:32,285 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,286 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:32,286 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001/tez-conf.pb" } size: 36913 timestamp: 1463533052000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:32,286 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,287 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:32,287 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:32,288 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533052126_0001_1, initTime=1463533052287
2016-05-18 00:57:32,288 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533052126_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:32,288 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:32,288 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:DAG_STARTED]: dagID=dag_1463533052126_0001_1, startTime=1463533052288
2016-05-18 00:57:32,288 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533052126_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:32,289 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A780BBAEFCB8491696D53C465F37FA5A : {AAE5378E717B433B8AE77B6E7B4B9DA8={InputName=AAE5378E717B433B8AE77B6E7B4B9DA8}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:32,289 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: AAE5378E717B433B8AE77B6E7B4B9DA8, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:32,289 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,289 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,289 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,289 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,289 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...latform/-1_VALUES_REPLACE, applicationId=application_1463533052126_0001, dagName=[D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700] every/(1/1) ...latform/-1_VALUES_REPLACE
2016-05-18 00:57:32,290 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:32,290 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,292 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: AAE5378E717B433B8AE77B6E7B4B9DA8 on vertex vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,294 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:32,294 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,294 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,294 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input AAE5378E717B433B8AE77B6E7B4B9DA8 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:32,295 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533052126_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:32,295 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57], numAdditionalOutputs=1
2016-05-18 00:57:32,295 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=54474D02423B4C81B873316D934889DC, vertex=vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:32,297 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:32,297 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:32,297 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DD1C3EC697D845C19F138400933CFD57:54474D02423B4C81B873316D934889DC using oldmapred API
2016-05-18 00:57:32,297 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=54474D02423B4C81B873316D934889DC, vertexName=DD1C3EC697D845C19F138400933CFD57, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:32,298 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DD1C3EC697D845C19F138400933CFD57, vertexId=vertex_1463533052126_0001_1_01, initRequestedTime=1463533052290, initedTime=1463533052295, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:32,297 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:32,298 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:32,299 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:32,299 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: AAE5378E717B433B8AE77B6E7B4B9DA8 on vertex vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,300 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:32,300 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {AAE5378E717B433B8AE77B6E7B4B9DA8=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:32,301 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] parallelism set to 1
2016-05-18 00:57:32,301 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:32,301 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A780BBAEFCB8491696D53C465F37FA5A, vertexId=vertex_1463533052126_0001_1_00, initRequestedTime=1463533052289, initedTime=1463533052301, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:32,301 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] with distanceFromRoot: 0
2016-05-18 00:57:32,302 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:32,302 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533052126_0001_1_00, startRequestedTime=1463533052290, startedTime=1463533052302
2016-05-18 00:57:32,302 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533052126_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:32,302 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:32,302 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533052126_0001_1_00 for vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] numStartedSources: 1 numSources: 1
2016-05-18 00:57:32,302 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] with distanceFromRoot: 1
2016-05-18 00:57:32,303 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533052126_0001_1_01, startRequestedTime=1463533052302, startedTime=1463533052302
2016-05-18 00:57:32,303 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:32,306 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DD1C3EC697D845C19F138400933CFD57 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:32,308 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A780BBAEFCB8491696D53C465F37FA5A
2016-05-18 00:57:32,308 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,308 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:TASK_STARTED]: vertexName=A780BBAEFCB8491696D53C465F37FA5A, taskId=task_1463533052126_0001_1_00_000000, scheduledTime=1463533052308, launchTime=1463533052308
2016-05-18 00:57:32,308 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,309 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A780BBAEFCB8491696D53C465F37FA5A in vertex: DD1C3EC697D845C19F138400933CFD57 numBipartiteSourceTasks: 1
2016-05-18 00:57:32,309 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:32,311 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533052126_0001_00_000001, launchTime=1463533052311
2016-05-18 00:57:32,311 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533052126_0001_00_000001
2016-05-18 00:57:32,312 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533052126_0001_1_00_000000_0] started. Is using containerId: [container_1463533052126_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:32,312 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A780BBAEFCB8491696D53C465F37FA5A, taskAttemptId=attempt_1463533052126_0001_1_00_000000_0, startTime=1463533052312, containerId=container_1463533052126_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:32,313 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533052126_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533052126_0001_1_00_000000_0
2016-05-18 00:57:32 Starting to run new task attempt: attempt_1463533052126_0001_1_00_000000_0
2016-05-18 00:57:32,313 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533052126_0001_1_00_000000_0
2016-05-18 00:57:32,313 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,313 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:32,314 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700] every/(1/1) ...latform/-1_VALUES_REPLACE, VertexName: A780BBAEFCB8491696D53C465F37FA5A, VertexParallelism: 1, TaskAttemptID:attempt_1463533052126_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AAE5378E717B433B8AE77B6E7B4B9DA8, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DD1C3EC697D845C19F138400933CFD57, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:32,314 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:32,314 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533052126_0001_1_00_000000_0
2016-05-18 00:57:32,315 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533052126_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:32,316 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533052126_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:32,317 INFO input.MRInput (MRInput.java:initialize(440)) - AAE5378E717B433B8AE77B6E7B4B9DA8 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:32,317 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: AAE5378E717B433B8AE77B6E7B4B9DA8
2016-05-18 00:57:32,318 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:32,321 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533052126_0001, with dag id: dag_1463533052126_0001_1
2016-05-18 00:57:32,325 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A780BBAEFCB8491696D53C465F37FA5A, ordinal: 0
2016-05-18 00:57:32,326 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A780BBAEFCB8491696D53C465F37FA5A, mem on start (mb), free: 125, total: 365, max: 672
2016-05-18 00:57:32,326 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:32,326 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:32,326 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:32,327 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:32,327 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[AAE5378E717B433B8AE77B6E7B4B9DA8:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [DD1C3EC697D845C19F138400933CFD57:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:32,327 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:32,327 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:32,328 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:32,328 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[AAE5378E717B433B8AE77B6E7B4B9DA8:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [DD1C3EC697D845C19F138400933CFD57:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:32,328 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:32,328 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AAE5378E717B433B8AE77B6E7B4B9DA8 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:32,328 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:32,329 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:32,330 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AAE5378E717B433B8AE77B6E7B4B9DA8
2016-05-18 00:57:32,330 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:32,331 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533052126_0001_1_00_000000_0
2016-05-18 00:57:32,338 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:32,340 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: AAE5378E717B433B8AE77B6E7B4B9DA8
2016-05-18 00:57:32,340 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: BC79D68EF669433FB9CE978BD2EAA6B3
2016-05-18 00:57:32,341 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] BC79D68EF669433FB9CE978BD2EAA6B3
2016-05-18 00:57:32,339 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:32,342 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - AAE5378E717B433B8AE77B6E7B4B9DA8 initialized RecordReader from event
2016-05-18 00:57:32,342 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:32,344 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:32,344 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DD1C3EC697D845C19F138400933CFD57 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@56eabf4a, comparator=cascading.tuple.hadoop.util.TupleComparator@4db4bfda, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:32,345 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DD1C3EC697D845C19F138400933CFD57: , UsingHashComparator=false
2016-05-18 00:57:32,375 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:32,375 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:32,375 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DD1C3EC697D845C19F138400933CFD57: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:32,375 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DD1C3EC697D845C19F138400933CFD57: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:32,376 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:32,376 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A780BBAEFCB8491696D53C465F37FA5A, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:32,380 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533052126_0001_1_00_000000_0
2016-05-18 00:57:32,380 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A780BBAEFCB8491696D53C465F37FA5A, mem on close (mb), free: 226, total: 364, max: 672
2016-05-18 00:57:32,380 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DD1C3EC697D845C19F138400933CFD57: Starting flush of map output
2016-05-18 00:57:32,380 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DD1C3EC697D845C19F138400933CFD57: Span0.length = 20, perItem = 6
2016-05-18 00:57:32,381 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DD1C3EC697D845C19F138400933CFD57: done sorting span=0, length=20, time=0
2016-05-18 00:57:32,381 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DD1C3EC697D845C19F138400933CFD57: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:32,383 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DD1C3EC697D845C19F138400933CFD57: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001_wd/localmode-local-dir/output/attempt_1463533052126_0001_1_00_000000_0_10081_0/file.out
2016-05-18 00:57:32,386 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:32 Completed running task attempt: attempt_1463533052126_0001_1_00_000000_0
2016-05-18 00:57:32,387 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533052126_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:32,387 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533052126_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:32,388 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533052126_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533052376, Process_Duration=4, Process_End_Time=1463533052380, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:32,389 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533052126_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:32,389 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:32,389 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A780BBAEFCB8491696D53C465F37FA5A, taskAttemptId=attempt_1463533052126_0001_1_00_000000_0, creationTime=1463533052308, allocationTime=1463533052310, startTime=1463533052312, finishTime=1463533052389, timeTaken=77, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:32,390 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533052126_0001_1_00_000000_0
2016-05-18 00:57:32,390 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:TASK_FINISHED]: vertexName=A780BBAEFCB8491696D53C465F37FA5A, taskId=task_1463533052126_0001_1_00_000000, startTime=1463533052312, finishTime=1463533052390, timeTaken=78, status=SUCCEEDED, successfulAttemptID=attempt_1463533052126_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533052376, Process_Duration=4, Process_End_Time=1463533052380, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:32,389 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:32,391 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,391 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:32,391 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A]
2016-05-18 00:57:32,392 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533052126_0001_00_000001
2016-05-18 00:57:32,392 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:VERTEX_FINISHED]: vertexName=A780BBAEFCB8491696D53C465F37FA5A, vertexId=vertex_1463533052126_0001_1_00, initRequestedTime=1463533052289, initedTime=1463533052301, startRequestedTime=1463533052290, startedTime=1463533052302, finishTime=1463533052392, timeTaken=90, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533052376, Process_Duration=4, Process_End_Time=1463533052380, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533052312, firstTasksToStart=[ task_1463533052126_0001_1_00_000000 ], lastTaskFinishTime=1463533052389, lastTasksToFinish=[ task_1463533052126_0001_1_00_000000 ], minTaskDuration=77, maxTaskDuration=77, avgTaskDuration=77.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533052126_0001_1_00_000000 ], longestDurationTasks=[ task_1463533052126_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:32,393 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:32,393 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] attempt: attempt_1463533052126_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:32,393 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DD1C3EC697D845C19F138400933CFD57
2016-05-18 00:57:32,393 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533052126_0001_00_000001, stoppedTime=1463533052393, exitStatus=0
2016-05-18 00:57:32,393 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A780BBAEFCB8491696D53C465F37FA5A, destinationVertexName=DD1C3EC697D845C19F138400933CFD57. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:32,394 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,393 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533052126_0001_1_00 [A780BBAEFCB8491696D53C465F37FA5A] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:32,394 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:32,394 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533052126_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:32,394 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:TASK_STARTED]: vertexName=DD1C3EC697D845C19F138400933CFD57, taskId=task_1463533052126_0001_1_01_000000, scheduledTime=1463533052394, launchTime=1463533052394
2016-05-18 00:57:32,395 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,397 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533052126_0001_00_000002, launchTime=1463533052397
2016-05-18 00:57:32,398 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533052126_0001_00_000002
2016-05-18 00:57:32,398 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533052126_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533052126_0001_1_01_000000_0
2016-05-18 00:57:32,399 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533052126_0001_1_01_000000_0] started. Is using containerId: [container_1463533052126_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:32,399 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DD1C3EC697D845C19F138400933CFD57, taskAttemptId=attempt_1463533052126_0001_1_01_000000_0, startTime=1463533052398, containerId=container_1463533052126_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:32 Starting to run new task attempt: attempt_1463533052126_0001_1_01_000000_0
2016-05-18 00:57:32,400 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533052126_0001_1_01_000000_0
2016-05-18 00:57:32,400 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,400 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:32,400 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700] every/(1/1) ...latform/-1_VALUES_REPLACE, VertexName: DD1C3EC697D845C19F138400933CFD57, VertexParallelism: 1, TaskAttemptID:attempt_1463533052126_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A780BBAEFCB8491696D53C465F37FA5A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=54474D02423B4C81B873316D934889DC, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:32,401 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:32,401 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533052126_0001_1_01_000000_0
2016-05-18 00:57:32,402 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533052126_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,402 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533052126_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:32,402 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533052126_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:32,403 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533052126_0001_00_000001, Exiting
2016-05-18 00:57:32,403 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533052126_0001_00_000001
2016-05-18 00:57:32,403 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533052126_0001_00_000001
2016-05-18 00:57:32,403 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533052126_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:32,407 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:32,410 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533052126_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:32,410 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533052126_0001_00_000001 completed successfully
2016-05-18 00:57:32,411 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:32,416 INFO output.MROutput (MROutput.java:initialize(421)) - 54474D02423B4C81B873316D934889DC: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:32,417 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DD1C3EC697D845C19F138400933CFD57, ordinal: 1
2016-05-18 00:57:32,417 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DD1C3EC697D845C19F138400933CFD57, mem on start (mb), free: 220, total: 364, max: 672
2016-05-18 00:57:32,417 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:32,418 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:32,418 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:32,419 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:32,419 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[54474D02423B4C81B873316D934889DC:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A780BBAEFCB8491696D53C465F37FA5A:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:32,420 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:32,420 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:32,420 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:32,424 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[54474D02423B4C81B873316D934889DC:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A780BBAEFCB8491696D53C465F37FA5A:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:32,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:32,425 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A780BBAEFCB8491696D53C465F37FA5A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:32,426 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:32,426 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:32,425 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A780BBAEFCB8491696D53C465F37FA5A: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:32,427 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A780BBAEFCB8491696D53C465F37FA5A: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:32,428 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A780BBAEFCB8491696D53C465F37FA5A with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:32,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A780BBAEFCB8491696D53C465F37FA5A
2016-05-18 00:57:32,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:32,429 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533052126_0001_1_01_000000_0
2016-05-18 00:57:32,430 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A780BBAEFCB8491696D53C465F37FA5A: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:32,432 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533052126_0001_1_00_000000_0_10081} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533052432, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:32,432 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A780BBAEFCB8491696D53C465F37FA5A
2016-05-18 00:57:32,432 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:32,433 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A780BBAEFCB8491696D53C465F37FA5A, wasInterrupted=false
2016-05-18 00:57:32,433 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:32,434 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:32,434 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:32,434 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: BC79D68EF669433FB9CE978BD2EAA6B3
2016-05-18 00:57:32,435 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_REPLACE"], id: 54474D02423B4C81B873316D934889DC
2016-05-18 00:57:32,435 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_REPLACE"]
2016-05-18 00:57:32,435 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:32,435 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] BC79D68EF669433FB9CE978BD2EAA6B3, for 1 inputs
2016-05-18 00:57:32,435 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001_wd/localmode-local-dir/output/attempt_1463533052126_0001_1_00_000000_0_10081/file.out, len=181, isLocal=true
2016-05-18 00:57:32,435 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:32,435 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:32,435 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:32,436 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - A780BBAEFCB8491696D53C465F37FA5A: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:32,436 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:32,436 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A780BBAEFCB8491696D53C465F37FA5A
2016-05-18 00:57:32,436 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A780BBAEFCB8491696D53C465F37FA5A: Shuffle Runner thread complete
2016-05-18 00:57:32,437 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DD1C3EC697D845C19F138400933CFD57, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:57:32,438 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:32,438 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A780BBAEFCB8491696D53C465F37FA5A: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:32,440 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:32,441 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:32,442 INFO output.MROutput (MROutput.java:close(526)) - 54474D02423B4C81B873316D934889DC closed
2016-05-18 00:57:32,442 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533052126_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:32,443 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533052126_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:32,443 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533052126_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533052437, Process_Duration=6, Process_End_Time=1463533052443, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:32,443 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:32,444 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DD1C3EC697D845C19F138400933CFD57, mem on close (mb), free: 217, total: 364, max: 672
2016-05-18 00:57:32,444 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=DD1C3EC697D845C19F138400933CFD57, index=1, interruptedStatus=false
2016-05-18 00:57:32,444 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A780BBAEFCB8491696D53C465F37FA5A
2016-05-18 00:57:32,444 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A780BBAEFCB8491696D53C465F37FA5A: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:32,445 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=DD1C3EC697D845C19F138400933CFD57, sourceVertex=A780BBAEFCB8491696D53C465F37FA5A, interruptedStatus=true
2016-05-18 00:57:32,445 INFO output.MROutput (MROutput.java:close(526)) - 54474D02423B4C81B873316D934889DC closed
2016-05-18 00:57:32,445 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=DD1C3EC697D845C19F138400933CFD57, sourceVertex=54474D02423B4C81B873316D934889DC, interruptedStatus=true
2016-05-18 00:57:32,446 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533052126_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:32,446 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:32,446 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533052126_0001_00_000002
2016-05-18 00:57:32,446 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533052126_0001_00_000002
2016-05-18 00:57:32,446 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DD1C3EC697D845C19F138400933CFD57, taskAttemptId=attempt_1463533052126_0001_1_01_000000_0, creationTime=1463533052395, allocationTime=1463533052396, startTime=1463533052398, finishTime=1463533052446, timeTaken=48, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533052126_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533052437, Process_Duration=6, Process_End_Time=1463533052443, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:32,447 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533052126_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:32,447 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533052126_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:32,447 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533052126_0001_00_000002 completed but with errors
2016-05-18 00:57:32,447 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:TASK_FINISHED]: vertexName=DD1C3EC697D845C19F138400933CFD57, taskId=task_1463533052126_0001_1_01_000000, startTime=1463533052398, finishTime=1463533052447, timeTaken=49, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533052126_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:32,447 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533052126_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:32,448 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533052126_0001_00_000002, stoppedTime=1463533052448, exitStatus=1
2016-05-18 00:57:32,448 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] because task failed: task_1463533052126_0001_1_01_000000
2016-05-18 00:57:32,448 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:32,448 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:32,448 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:32,448 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,448 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=54474D02423B4C81B873316D934889DC, vertexId=vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,450 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:VERTEX_FINISHED]: vertexName=DD1C3EC697D845C19F138400933CFD57, vertexId=vertex_1463533052126_0001_1_01, initRequestedTime=1463533052290, initedTime=1463533052295, startRequestedTime=1463533052302, startedTime=1463533052302, finishTime=1463533052450, timeTaken=148, status=FAILED, diagnostics=Task failed, taskId=task_1463533052126_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533052126_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533052398, firstTasksToStart=[ task_1463533052126_0001_1_01_000000 ], lastTaskFinishTime=1463533052446, lastTasksToFinish=[ task_1463533052126_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:32,451 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:32,451 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:32,452 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:32,452 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:32,454 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533052126_0001_1, queueSize=0
2016-05-18 00:57:32,455 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052126_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533052126_0001_1, startTime=1463533052288, finishTime=1463533052454, timeTaken=166, status=FAILED, diagnostics=Vertex failed, vertexName=DD1C3EC697D845C19F138400933CFD57, vertexId=vertex_1463533052126_0001_1_01, diagnostics=[Task failed, taskId=task_1463533052126_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533052126_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=18, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533052376, Process_Duration=4, Process_End_Time=1463533052380, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:32,455 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533052126_0001_1_01 [DD1C3EC697D845C19F138400933CFD57]
2016-05-18 00:57:32,455 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533052126_0001_1 finished with state: FAILED
2016-05-18 00:57:32,455 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533052126_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:32,455 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533052126_0001_1_01_000000_0 failed: true container: container_1463533052126_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:32 Completed Dag: dag_1463533052126_0001_1
2016-05-18 00:57:32,456 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533052126_0001_1, dagState=FAILED
2016-05-18 00:57:32,457 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533052126_0001_1_post
2016-05-18 00:57:32,457 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,457 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:32,457 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:32,457 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700] every/(1/1) ...latform/-1_VALUES_REPLACE, with id=dag_1463533052126_0001_1
2016-05-18 00:57:32,458 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700] every/(1/1) ...latform/-1_VALUES_REPLACE, with id=dag_1463533052126_0001_1
2016-05-18 00:57:32,462 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...latform/-1_VALUES_REPLACE, applicationId=application_1463533052126_0001
2016-05-18 00:57:32,466 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:32,466 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:32,467 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:32,467 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:32,467 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:32,467 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:32,467 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:32,468 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:32,468 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:32,468 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:32,468 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:32,468 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:32,469 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:32,469 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:32,472 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D1503B4A9DA244E78D62853A938F4465/B5CBB3EAAA3A4C3A93BB1E44C5447700/.tez/application_1463533052126_0001
2016-05-18 00:57:32,473 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:32,473 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...latform/-1_VALUES_REPLACE
2016-05-18 00:57:32,474 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:32,474 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_REPLACE/_temporary
2016-05-18 00:57:32,474 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.367
2016-05-18 00:57:32,475 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_VALUES_REPLACE
2016-05-18 00:57:32,499 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:32,499 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:32,530 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:32,530 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:32,534 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.010
2016-05-18 00:57:32,541 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 00:57:32,542 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:32,542 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:32,542 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:32,558 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:32,564 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_SWAP
2016-05-18 00:57:32,614 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:32,614 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:32,614 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_SWAP"]
2016-05-18 00:57:32,615 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:32,615 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:32,615 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:32,616 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/-1_VALUES_SWAP
2016-05-18 00:57:32,616 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:32,629 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:32,633 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:32,634 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:32,634 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:32,643 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001 doesn't exist and is created
2016-05-18 00:57:32,658 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:32,658 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:32,659 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001_wd
2016-05-18 00:57:32,669 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533052634_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:32,669 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:32,670 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:32,671 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:32,671 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:32,671 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:32,671 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:32,672 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:32,674 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:32,674 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:32,674 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:32,674 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:32,675 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:32,675 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:32,675 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:32,675 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:32,676 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:32,676 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:32,676 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:32,676 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:32,678 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:32,679 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:32,679 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:32,679 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001_wd/localmode-log-dir
2016-05-18 00:57:32,680 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001_wd/localmode-log-dir/history.txt.appattempt_1463533052634_0001_000000, maxErrors=10
2016-05-18 00:57:32,680 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:32,680 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533052634_0001_000000, appSubmitTime=1463533052669, launchTime=1463533052669
2016-05-18 00:57:32,684 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:32,687 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:37796
2016-05-18 00:57:32,688 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533052634
2016-05-18 00:57:32,690 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533052634_0001_000000, startTime=1463533052690
2016-05-18 00:57:32,690 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:32,759 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:32,759 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:32,760 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/-1_VALUES_SWAP, applicationId=application_1463533052634_0001, dagName=[D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA] every/(1/1) ...lyplatform/-1_VALUES_SWAP
2016-05-18 00:57:32,777 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA] every/(1/1) ...lyplatform/-1_VALUES_SWAP
2016-05-18 00:57:32,777 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:32,778 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533052634_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001_wd/localmode-log-dir/dag_1463533052634_0001_1.dot
2016-05-18 00:57:32,778 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001_wd/localmode-log-dir/dag_1463533052634_0001_1-tez-dag.pb.txt
2016-05-18 00:57:32,785 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533052634_0001_1
2016-05-18 00:57:32,785 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,785 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA] every/(1/1) ...lyplatform/-1_VALUES_SWAP
2016-05-18 00:57:32 Running Dag: dag_1463533052634_0001_1
2016-05-18 00:57:32,794 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533052634_0001_1, submitTime=1463533052777
2016-05-18 00:57:32,795 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001/tez-conf.pb" } size: 36913 timestamp: 1463533052000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:32,795 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,795 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:32,796 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001/tez-conf.pb" } size: 36913 timestamp: 1463533052000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:32,796 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,796 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:32,797 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:32,797 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533052634_0001_1, initTime=1463533052797
2016-05-18 00:57:32,797 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533052634_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:32,797 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:32,798 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:DAG_STARTED]: dagID=dag_1463533052634_0001_1, startTime=1463533052798
2016-05-18 00:57:32,798 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533052634_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:32,798 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B2F4891F16DD4470AD46282A1E838DA5 : {BA04E82CE28C4916AA3EAF58296420B7={InputName=BA04E82CE28C4916AA3EAF58296420B7}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:32,798 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: BA04E82CE28C4916AA3EAF58296420B7, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:32,798 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,798 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,799 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,799 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,800 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:32,800 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/-1_VALUES_SWAP, applicationId=application_1463533052634_0001, dagName=[D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA] every/(1/1) ...lyplatform/-1_VALUES_SWAP
2016-05-18 00:57:32,800 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,800 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: BA04E82CE28C4916AA3EAF58296420B7 on vertex vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,802 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input BA04E82CE28C4916AA3EAF58296420B7 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:32,804 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:32,804 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:32,806 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:32,806 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:32,806 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: BA04E82CE28C4916AA3EAF58296420B7 on vertex vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,802 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:32,807 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,807 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,807 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533052634_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:32,807 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073], numAdditionalOutputs=1
2016-05-18 00:57:32,807 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=787DBD1DA71C4FFABA7FA25D07C092B4, vertex=vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:32,809 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E8F243F8AB924A469040E7E02C330073:787DBD1DA71C4FFABA7FA25D07C092B4 using oldmapred API
2016-05-18 00:57:32,810 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=787DBD1DA71C4FFABA7FA25D07C092B4, vertexName=E8F243F8AB924A469040E7E02C330073, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:32,811 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E8F243F8AB924A469040E7E02C330073, vertexId=vertex_1463533052634_0001_1_01, initRequestedTime=1463533052800, initedTime=1463533052807, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:32,811 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:32,812 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:32,812 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {BA04E82CE28C4916AA3EAF58296420B7=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:32,812 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] parallelism set to 1
2016-05-18 00:57:32,813 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:32,813 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B2F4891F16DD4470AD46282A1E838DA5, vertexId=vertex_1463533052634_0001_1_00, initRequestedTime=1463533052798, initedTime=1463533052813, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:32,813 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] with distanceFromRoot: 0
2016-05-18 00:57:32,813 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:32,814 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533052634_0001_1_00, startRequestedTime=1463533052800, startedTime=1463533052814
2016-05-18 00:57:32,814 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533052634_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:32,814 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:32,815 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B2F4891F16DD4470AD46282A1E838DA5
2016-05-18 00:57:32,815 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533052634_0001_1_00 for vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] numStartedSources: 1 numSources: 1
2016-05-18 00:57:32,815 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] with distanceFromRoot: 1
2016-05-18 00:57:32,815 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,816 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533052634_0001_1_01, startRequestedTime=1463533052815, startedTime=1463533052815
2016-05-18 00:57:32,816 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:32,817 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E8F243F8AB924A469040E7E02C330073 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:32,817 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:TASK_STARTED]: vertexName=B2F4891F16DD4470AD46282A1E838DA5, taskId=task_1463533052634_0001_1_00_000000, scheduledTime=1463533052817, launchTime=1463533052817
2016-05-18 00:57:32,817 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B2F4891F16DD4470AD46282A1E838DA5 in vertex: E8F243F8AB924A469040E7E02C330073 numBipartiteSourceTasks: 1
2016-05-18 00:57:32,818 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,819 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:32,820 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533052634_0001, with dag id: dag_1463533052634_0001_1
2016-05-18 00:57:32,821 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533052634_0001_00_000001, launchTime=1463533052821
2016-05-18 00:57:32,822 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533052634_0001_00_000001
2016-05-18 00:57:32,825 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533052634_0001_1_00_000000_0] started. Is using containerId: [container_1463533052634_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:32,825 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B2F4891F16DD4470AD46282A1E838DA5, taskAttemptId=attempt_1463533052634_0001_1_00_000000_0, startTime=1463533052825, containerId=container_1463533052634_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:32,826 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533052634_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533052634_0001_1_00_000000_0
2016-05-18 00:57:32 Starting to run new task attempt: attempt_1463533052634_0001_1_00_000000_0
2016-05-18 00:57:32,827 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533052634_0001_1_00_000000_0
2016-05-18 00:57:32,827 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,827 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:32,827 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA] every/(1/1) ...lyplatform/-1_VALUES_SWAP, VertexName: B2F4891F16DD4470AD46282A1E838DA5, VertexParallelism: 1, TaskAttemptID:attempt_1463533052634_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BA04E82CE28C4916AA3EAF58296420B7, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E8F243F8AB924A469040E7E02C330073, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:32,827 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:32,828 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533052634_0001_1_00_000000_0
2016-05-18 00:57:32,828 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533052634_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:32,828 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533052634_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:32,832 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:32,832 INFO input.MRInput (MRInput.java:initialize(440)) - BA04E82CE28C4916AA3EAF58296420B7 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:32,833 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: BA04E82CE28C4916AA3EAF58296420B7
2016-05-18 00:57:32,837 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B2F4891F16DD4470AD46282A1E838DA5, ordinal: 0
2016-05-18 00:57:32,837 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B2F4891F16DD4470AD46282A1E838DA5, mem on start (mb), free: 133, total: 364, max: 672
2016-05-18 00:57:32,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:32,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:32,837 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:32,838 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:32,838 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BA04E82CE28C4916AA3EAF58296420B7:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E8F243F8AB924A469040E7E02C330073:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:32,838 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:32,838 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:32,838 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:32,838 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BA04E82CE28C4916AA3EAF58296420B7:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E8F243F8AB924A469040E7E02C330073:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:32,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:32,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BA04E82CE28C4916AA3EAF58296420B7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:32,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:32,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:32,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BA04E82CE28C4916AA3EAF58296420B7
2016-05-18 00:57:32,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:32,841 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533052634_0001_1_00_000000_0
2016-05-18 00:57:32,843 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:32,843 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - BA04E82CE28C4916AA3EAF58296420B7 initialized RecordReader from event
2016-05-18 00:57:32,845 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:32,845 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: BA04E82CE28C4916AA3EAF58296420B7
2016-05-18 00:57:32,845 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: A01942FEAB694D4C95138E2732FFE466
2016-05-18 00:57:32,845 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] A01942FEAB694D4C95138E2732FFE466
2016-05-18 00:57:32,847 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:32,849 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:32,849 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E8F243F8AB924A469040E7E02C330073 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@31813560, comparator=cascading.tuple.hadoop.util.TupleComparator@19d74b04, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:32,850 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E8F243F8AB924A469040E7E02C330073: , UsingHashComparator=false
2016-05-18 00:57:32,876 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:32,876 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:32,877 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E8F243F8AB924A469040E7E02C330073: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:32,877 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E8F243F8AB924A469040E7E02C330073: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:32,877 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:32,877 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B2F4891F16DD4470AD46282A1E838DA5, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:32,880 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533052634_0001_1_00_000000_0
2016-05-18 00:57:32,881 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B2F4891F16DD4470AD46282A1E838DA5, mem on close (mb), free: 222, total: 361, max: 672
2016-05-18 00:57:32,881 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E8F243F8AB924A469040E7E02C330073: Starting flush of map output
2016-05-18 00:57:32,881 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E8F243F8AB924A469040E7E02C330073: Span0.length = 20, perItem = 6
2016-05-18 00:57:32,882 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E8F243F8AB924A469040E7E02C330073: done sorting span=0, length=20, time=1
2016-05-18 00:57:32,882 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E8F243F8AB924A469040E7E02C330073: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:32,883 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E8F243F8AB924A469040E7E02C330073: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001_wd/localmode-local-dir/output/attempt_1463533052634_0001_1_00_000000_0_10086_0/file.out
2016-05-18 00:57:32,888 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:32 Completed running task attempt: attempt_1463533052634_0001_1_00_000000_0
2016-05-18 00:57:32,889 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533052634_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:32,889 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533052634_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:32,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533052634_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533052878, Process_Duration=2, Process_End_Time=1463533052880, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:32,891 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533052634_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:32,891 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:32,891 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:32,891 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B2F4891F16DD4470AD46282A1E838DA5, taskAttemptId=attempt_1463533052634_0001_1_00_000000_0, creationTime=1463533052818, allocationTime=1463533052821, startTime=1463533052825, finishTime=1463533052892, timeTaken=67, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:32,892 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533052634_0001_1_00_000000_0
2016-05-18 00:57:32,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:TASK_FINISHED]: vertexName=B2F4891F16DD4470AD46282A1E838DA5, taskId=task_1463533052634_0001_1_00_000000, startTime=1463533052825, finishTime=1463533052892, timeTaken=67, status=SUCCEEDED, successfulAttemptID=attempt_1463533052634_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533052878, Process_Duration=2, Process_End_Time=1463533052880, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:32,893 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:32,893 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5]
2016-05-18 00:57:32,896 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:VERTEX_FINISHED]: vertexName=B2F4891F16DD4470AD46282A1E838DA5, vertexId=vertex_1463533052634_0001_1_00, initRequestedTime=1463533052798, initedTime=1463533052813, startRequestedTime=1463533052800, startedTime=1463533052814, finishTime=1463533052893, timeTaken=79, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533052878, Process_Duration=2, Process_End_Time=1463533052880, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533052825, firstTasksToStart=[ task_1463533052634_0001_1_00_000000 ], lastTaskFinishTime=1463533052892, lastTasksToFinish=[ task_1463533052634_0001_1_00_000000 ], minTaskDuration=67, maxTaskDuration=67, avgTaskDuration=67.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533052634_0001_1_00_000000 ], longestDurationTasks=[ task_1463533052634_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:32,896 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:32,896 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] attempt: attempt_1463533052634_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:32,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533052634_0001_00_000001, stoppedTime=1463533052896, exitStatus=0
2016-05-18 00:57:32,897 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533052634_0001_1_00 [B2F4891F16DD4470AD46282A1E838DA5] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:32,897 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:32,897 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E8F243F8AB924A469040E7E02C330073
2016-05-18 00:57:32,897 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533052634_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:32,897 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B2F4891F16DD4470AD46282A1E838DA5, destinationVertexName=E8F243F8AB924A469040E7E02C330073. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:32,898 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,899 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:TASK_STARTED]: vertexName=E8F243F8AB924A469040E7E02C330073, taskId=task_1463533052634_0001_1_01_000000, scheduledTime=1463533052899, launchTime=1463533052899
2016-05-18 00:57:32,899 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,900 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533052634_0001_00_000001
2016-05-18 00:57:32,901 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533052634_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:32,902 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533052634_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:32,902 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533052634_0001_00_000001, Exiting
2016-05-18 00:57:32,902 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533052634_0001_00_000001
2016-05-18 00:57:32,902 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533052634_0001_00_000001
2016-05-18 00:57:32,901 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533052634_0001_00_000002, launchTime=1463533052901
2016-05-18 00:57:32,904 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533052634_0001_00_000002
2016-05-18 00:57:32,909 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533052634_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:32,909 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533052634_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533052634_0001_1_01_000000_0
2016-05-18 00:57:32,909 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533052634_0001_1_01_000000_0] started. Is using containerId: [container_1463533052634_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:32,911 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E8F243F8AB924A469040E7E02C330073, taskAttemptId=attempt_1463533052634_0001_1_01_000000_0, startTime=1463533052909, containerId=container_1463533052634_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:32,910 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533052634_0001_00_000001 completed successfully
2016-05-18 00:57:32 Starting to run new task attempt: attempt_1463533052634_0001_1_01_000000_0
2016-05-18 00:57:32,911 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533052634_0001_1_01_000000_0
2016-05-18 00:57:32,911 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,912 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:32,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA] every/(1/1) ...lyplatform/-1_VALUES_SWAP, VertexName: E8F243F8AB924A469040E7E02C330073, VertexParallelism: 1, TaskAttemptID:attempt_1463533052634_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B2F4891F16DD4470AD46282A1E838DA5, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=787DBD1DA71C4FFABA7FA25D07C092B4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:32,912 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:32,913 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533052634_0001_1_01_000000_0
2016-05-18 00:57:32,915 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533052634_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,915 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533052634_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:32,916 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:32,919 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:32,926 INFO output.MROutput (MROutput.java:initialize(421)) - 787DBD1DA71C4FFABA7FA25D07C092B4: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:32,929 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E8F243F8AB924A469040E7E02C330073, ordinal: 1
2016-05-18 00:57:32,929 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E8F243F8AB924A469040E7E02C330073, mem on start (mb), free: 216, total: 361, max: 672
2016-05-18 00:57:32,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:32,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:32,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:32,929 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:32,930 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B2F4891F16DD4470AD46282A1E838DA5:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [787DBD1DA71C4FFABA7FA25D07C092B4:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:32,930 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:32,930 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:32,931 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:32,931 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B2F4891F16DD4470AD46282A1E838DA5:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [787DBD1DA71C4FFABA7FA25D07C092B4:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:32,931 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:32,931 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B2F4891F16DD4470AD46282A1E838DA5 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:32,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:32,932 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:32,932 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B2F4891F16DD4470AD46282A1E838DA5: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:32,933 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B2F4891F16DD4470AD46282A1E838DA5: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:32,934 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B2F4891F16DD4470AD46282A1E838DA5 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:32,934 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B2F4891F16DD4470AD46282A1E838DA5
2016-05-18 00:57:32,935 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:32,938 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533052634_0001_1_01_000000_0
2016-05-18 00:57:32,942 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B2F4891F16DD4470AD46282A1E838DA5: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:32,942 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:32,943 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: A01942FEAB694D4C95138E2732FFE466
2016-05-18 00:57:32,943 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533052634_0001_1_00_000000_0_10086} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533052943, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:32,944 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B2F4891F16DD4470AD46282A1E838DA5
2016-05-18 00:57:32,943 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_SWAP"], id: 787DBD1DA71C4FFABA7FA25D07C092B4
2016-05-18 00:57:32,944 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:32,944 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_SWAP"]
2016-05-18 00:57:32,944 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B2F4891F16DD4470AD46282A1E838DA5, wasInterrupted=false
2016-05-18 00:57:32,945 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] A01942FEAB694D4C95138E2732FFE466, for 1 inputs
2016-05-18 00:57:32,945 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:32,946 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:32,947 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:32,947 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001_wd/localmode-local-dir/output/attempt_1463533052634_0001_1_00_000000_0_10086/file.out, len=181, isLocal=true
2016-05-18 00:57:32,947 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:32,947 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:32,947 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:32,948 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:32,948 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B2F4891F16DD4470AD46282A1E838DA5
2016-05-18 00:57:32,948 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B2F4891F16DD4470AD46282A1E838DA5: Shuffle Runner thread complete
2016-05-18 00:57:32,948 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E8F243F8AB924A469040E7E02C330073, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:32,950 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:32,951 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B2F4891F16DD4470AD46282A1E838DA5: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:32,955 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:32,955 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:32,956 INFO output.MROutput (MROutput.java:close(526)) - 787DBD1DA71C4FFABA7FA25D07C092B4 closed
2016-05-18 00:57:32,957 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533052634_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:32,961 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533052634_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:32,962 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533052634_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533052949, Process_Duration=12, Process_End_Time=1463533052961, Read_Duration=1, Tuples_Read=1]]
2016-05-18 00:57:32,962 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:32,962 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E8F243F8AB924A469040E7E02C330073, mem on close (mb), free: 213, total: 361, max: 672
2016-05-18 00:57:32,963 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=E8F243F8AB924A469040E7E02C330073, index=1, interruptedStatus=false
2016-05-18 00:57:32,964 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B2F4891F16DD4470AD46282A1E838DA5
2016-05-18 00:57:32,964 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B2F4891F16DD4470AD46282A1E838DA5: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:32,964 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=E8F243F8AB924A469040E7E02C330073, sourceVertex=B2F4891F16DD4470AD46282A1E838DA5, interruptedStatus=true
2016-05-18 00:57:32,965 INFO output.MROutput (MROutput.java:close(526)) - 787DBD1DA71C4FFABA7FA25D07C092B4 closed
2016-05-18 00:57:32,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=E8F243F8AB924A469040E7E02C330073, sourceVertex=787DBD1DA71C4FFABA7FA25D07C092B4, interruptedStatus=true
2016-05-18 00:57:32,966 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533052634_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:32,966 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:32,966 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533052634_0001_00_000002
2016-05-18 00:57:32,966 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533052634_0001_00_000002
2016-05-18 00:57:32,966 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E8F243F8AB924A469040E7E02C330073, taskAttemptId=attempt_1463533052634_0001_1_01_000000_0, creationTime=1463533052899, allocationTime=1463533052900, startTime=1463533052909, finishTime=1463533052966, timeTaken=57, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533052634_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=10, LAST_EVENT_RECEIVED=10, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533052949, Process_Duration=12, Process_End_Time=1463533052961, Read_Duration=1, Tuples_Read=1
2016-05-18 00:57:32,967 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533052634_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:32,967 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:TASK_FINISHED]: vertexName=E8F243F8AB924A469040E7E02C330073, taskId=task_1463533052634_0001_1_01_000000, startTime=1463533052909, finishTime=1463533052967, timeTaken=58, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533052634_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:32,967 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] because task failed: task_1463533052634_0001_1_01_000000
2016-05-18 00:57:32,967 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:32,967 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:32,967 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:32,968 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,968 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=787DBD1DA71C4FFABA7FA25D07C092B4, vertexId=vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,970 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533052634_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:32,970 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533052634_0001_00_000002 completed but with errors
2016-05-18 00:57:32,970 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:VERTEX_FINISHED]: vertexName=E8F243F8AB924A469040E7E02C330073, vertexId=vertex_1463533052634_0001_1_01, initRequestedTime=1463533052800, initedTime=1463533052807, startRequestedTime=1463533052815, startedTime=1463533052815, finishTime=1463533052970, timeTaken=155, status=FAILED, diagnostics=Task failed, taskId=task_1463533052634_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533052634_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533052909, firstTasksToStart=[ task_1463533052634_0001_1_01_000000 ], lastTaskFinishTime=1463533052966, lastTasksToFinish=[ task_1463533052634_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:32,970 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:32,970 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533052634_0001_00_000002, stoppedTime=1463533052970, exitStatus=-1000
2016-05-18 00:57:32,971 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533052634_0001_1_01_000000_0 failed: true container: container_1463533052634_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:32,971 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:32,971 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:32,971 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:32,979 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533052634_0001_1, queueSize=0
2016-05-18 00:57:32,979 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533052634_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533052634_0001_1, startTime=1463533052797, finishTime=1463533052978, timeTaken=181, status=FAILED, diagnostics=Vertex failed, vertexName=E8F243F8AB924A469040E7E02C330073, vertexId=vertex_1463533052634_0001_1_01, diagnostics=[Task failed, taskId=task_1463533052634_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533052634_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=410, AM_GC_TIME_MILLIS=13, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533052878, Process_Duration=2, Process_End_Time=1463533052880, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:32,979 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533052634_0001_1_01 [E8F243F8AB924A469040E7E02C330073]
2016-05-18 00:57:32,979 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533052634_0001_1 finished with state: FAILED
2016-05-18 00:57:32,979 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533052634_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:32,980 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533052634_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:32,980 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533052634_0001_00_000002
2016-05-18 00:57:32 Completed Dag: dag_1463533052634_0001_1
2016-05-18 00:57:32,981 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533052634_0001_1, dagState=FAILED
2016-05-18 00:57:32,981 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533052634_0001_1_post
2016-05-18 00:57:32,982 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:32,982 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 2
2016-05-18 00:57:32,982 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:32,982 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA] every/(1/1) ...lyplatform/-1_VALUES_SWAP, with id=dag_1463533052634_0001_1
2016-05-18 00:57:32,983 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA] every/(1/1) ...lyplatform/-1_VALUES_SWAP, with id=dag_1463533052634_0001_1
2016-05-18 00:57:32,988 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/-1_VALUES_SWAP, applicationId=application_1463533052634_0001
2016-05-18 00:57:32,989 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:32,989 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:32,990 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:32,990 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:32,990 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:32,991 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:32,994 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:32,994 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:32,995 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:32,995 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:32,995 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:32,995 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:32,996 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:32,997 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:33,000 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D14FE89951654443B1506ECE37F7F0BF/D2B2FE2446AF40E29D0980EB0F2DDFDA/.tez/application_1463533052634_0001
2016-05-18 00:57:33,001 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:33,001 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...lyplatform/-1_VALUES_SWAP
2016-05-18 00:57:33,002 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:33,002 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_VALUES_SWAP/_temporary
2016-05-18 00:57:33,003 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.388
2016-05-18 00:57:33,003 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_VALUES_SWAP
2016-05-18 00:57:33,042 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:33,042 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:33,094 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 00:57:33,103 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.022
2016-05-18 00:57:33,103 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:33,103 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:33,103 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:33,119 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:33,126 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_-1
2016-05-18 00:57:33,174 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:33,175 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:33,175 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_-1"]
2016-05-18 00:57:33,175 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:33,175 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:33,176 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:33,176 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...semblyplatform/-1_ARGS_-1
2016-05-18 00:57:33,177 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:33,196 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:33,201 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:33,201 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:33,201 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:33,211 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001 doesn't exist and is created
2016-05-18 00:57:33,231 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:33,231 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:33,231 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:33,231 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:33,232 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001_wd
2016-05-18 00:57:33,244 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533053201_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:33,245 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:33,245 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:33,247 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:33,247 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:33,247 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:33,247 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:33,248 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:33,250 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:33,250 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:33,250 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:33,250 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:33,250 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:33,250 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:33,251 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:33,251 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:33,251 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:33,251 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:33,251 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:33,252 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:33,253 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:33,254 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:33,254 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:33,255 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001_wd/localmode-log-dir
2016-05-18 00:57:33,255 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001_wd/localmode-log-dir/history.txt.appattempt_1463533053201_0001_000000, maxErrors=10
2016-05-18 00:57:33,256 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:33,256 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533053201_0001_000000, appSubmitTime=1463533053244, launchTime=1463533053244
2016-05-18 00:57:33,259 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:33,265 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:43973
2016-05-18 00:57:33,266 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533053201
2016-05-18 00:57:33,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533053201_0001_000000, startTime=1463533053270
2016-05-18 00:57:33,270 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:33,331 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:33,332 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:33,332 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...semblyplatform/-1_ARGS_-1, applicationId=application_1463533053201_0001, dagName=[7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C] every/(1/1) ...semblyplatform/-1_ARGS_-1
2016-05-18 00:57:33,349 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C] every/(1/1) ...semblyplatform/-1_ARGS_-1
2016-05-18 00:57:33,350 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:33,351 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533053201_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001_wd/localmode-log-dir/dag_1463533053201_0001_1.dot
2016-05-18 00:57:33,352 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001_wd/localmode-log-dir/dag_1463533053201_0001_1-tez-dag.pb.txt
2016-05-18 00:57:33,367 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533053201_0001_1
2016-05-18 00:57:33,368 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:33,368 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C] every/(1/1) ...semblyplatform/-1_ARGS_-1
2016-05-18 00:57:33 Running Dag: dag_1463533053201_0001_1
2016-05-18 00:57:33,378 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533053201_0001_1, submitTime=1463533053350
2016-05-18 00:57:33,379 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001/tez-conf.pb" } size: 36913 timestamp: 1463533053000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:33,379 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,380 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:33,380 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001/tez-conf.pb" } size: 36913 timestamp: 1463533053000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:33,380 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]
2016-05-18 00:57:33,380 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:33,380 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:33,381 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533053201_0001_1, initTime=1463533053381
2016-05-18 00:57:33,381 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533053201_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:33,381 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:33,382 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:DAG_STARTED]: dagID=dag_1463533053201_0001_1, startTime=1463533053382
2016-05-18 00:57:33,382 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533053201_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:33,382 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B10EC62DB61A4427B4FA58319CB707C6 : {DB9A5CB9962A4D25B5C198C1647842DC={InputName=DB9A5CB9962A4D25B5C198C1647842DC}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:33,382 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: DB9A5CB9962A4D25B5C198C1647842DC, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:33,382 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,383 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,383 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,383 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,383 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...semblyplatform/-1_ARGS_-1, applicationId=application_1463533053201_0001, dagName=[7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C] every/(1/1) ...semblyplatform/-1_ARGS_-1
2016-05-18 00:57:33,384 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:33,384 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]
2016-05-18 00:57:33,384 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: DB9A5CB9962A4D25B5C198C1647842DC on vertex vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,386 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input DB9A5CB9962A4D25B5C198C1647842DC asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:33,387 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:33,387 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]
2016-05-18 00:57:33,388 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]
2016-05-18 00:57:33,388 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533053201_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:33,388 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF], numAdditionalOutputs=1
2016-05-18 00:57:33,388 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=881A5ED590074B72991F324054E4D823, vertex=vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:33,388 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:33,388 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:33,389 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:33,389 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:33,389 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: DB9A5CB9962A4D25B5C198C1647842DC on vertex vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,390 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D83293AA11544C1397049AC88FD8F2DF:881A5ED590074B72991F324054E4D823 using oldmapred API
2016-05-18 00:57:33,390 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=881A5ED590074B72991F324054E4D823, vertexName=D83293AA11544C1397049AC88FD8F2DF, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:33,391 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D83293AA11544C1397049AC88FD8F2DF, vertexId=vertex_1463533053201_0001_1_01, initRequestedTime=1463533053384, initedTime=1463533053388, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:33,391 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:33,392 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:33,392 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {DB9A5CB9962A4D25B5C198C1647842DC=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:33,393 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] parallelism set to 1
2016-05-18 00:57:33,393 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:33,393 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B10EC62DB61A4427B4FA58319CB707C6, vertexId=vertex_1463533053201_0001_1_00, initRequestedTime=1463533053382, initedTime=1463533053393, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:33,393 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] with distanceFromRoot: 0
2016-05-18 00:57:33,393 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:33,393 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533053201_0001_1_00, startRequestedTime=1463533053384, startedTime=1463533053393
2016-05-18 00:57:33,394 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533053201_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:33,394 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:33,394 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533053201_0001_1_00 for vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF] numStartedSources: 1 numSources: 1
2016-05-18 00:57:33,394 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF] with distanceFromRoot: 1
2016-05-18 00:57:33,394 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533053201_0001_1_01, startRequestedTime=1463533053394, startedTime=1463533053394
2016-05-18 00:57:33,394 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:33,395 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B10EC62DB61A4427B4FA58319CB707C6
2016-05-18 00:57:33,395 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,396 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:TASK_STARTED]: vertexName=B10EC62DB61A4427B4FA58319CB707C6, taskId=task_1463533053201_0001_1_00_000000, scheduledTime=1463533053396, launchTime=1463533053396
2016-05-18 00:57:33,397 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,397 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D83293AA11544C1397049AC88FD8F2DF with 0 source tasks and 1 pending tasks
2016-05-18 00:57:33,400 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B10EC62DB61A4427B4FA58319CB707C6 in vertex: D83293AA11544C1397049AC88FD8F2DF numBipartiteSourceTasks: 1
2016-05-18 00:57:33,400 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:33,402 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533053201_0001_00_000001, launchTime=1463533053401
2016-05-18 00:57:33,402 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533053201_0001_00_000001
2016-05-18 00:57:33,403 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533053201_0001_1_00_000000_0] started. Is using containerId: [container_1463533053201_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:33,403 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B10EC62DB61A4427B4FA58319CB707C6, taskAttemptId=attempt_1463533053201_0001_1_00_000000_0, startTime=1463533053403, containerId=container_1463533053201_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:33,403 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533053201_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533053201_0001_1_00_000000_0
2016-05-18 00:57:33 Starting to run new task attempt: attempt_1463533053201_0001_1_00_000000_0
2016-05-18 00:57:33,404 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533053201_0001_1_00_000000_0
2016-05-18 00:57:33,404 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:33,404 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:33,404 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C] every/(1/1) ...semblyplatform/-1_ARGS_-1, VertexName: B10EC62DB61A4427B4FA58319CB707C6, VertexParallelism: 1, TaskAttemptID:attempt_1463533053201_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DB9A5CB9962A4D25B5C198C1647842DC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D83293AA11544C1397049AC88FD8F2DF, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:33,405 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:33,405 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533053201_0001_1_00_000000_0
2016-05-18 00:57:33,406 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533053201_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:33,406 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533053201_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:33,411 INFO input.MRInput (MRInput.java:initialize(440)) - DB9A5CB9962A4D25B5C198C1647842DC using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:33,412 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:33,412 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: DB9A5CB9962A4D25B5C198C1647842DC
2016-05-18 00:57:33,417 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533053201_0001, with dag id: dag_1463533053201_0001_1
2016-05-18 00:57:33,420 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B10EC62DB61A4427B4FA58319CB707C6, ordinal: 0
2016-05-18 00:57:33,421 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B10EC62DB61A4427B4FA58319CB707C6, mem on start (mb), free: 127, total: 361, max: 672
2016-05-18 00:57:33,421 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:33,422 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:33,422 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:33,422 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:33,422 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DB9A5CB9962A4D25B5C198C1647842DC:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D83293AA11544C1397049AC88FD8F2DF:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:33,424 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:33,424 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:33,425 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:33,426 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DB9A5CB9962A4D25B5C198C1647842DC:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D83293AA11544C1397049AC88FD8F2DF:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:33,426 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:33,426 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DB9A5CB9962A4D25B5C198C1647842DC
2016-05-18 00:57:33,427 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DB9A5CB9962A4D25B5C198C1647842DC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:33,427 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:33,427 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:33,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:33,428 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533053201_0001_1_00_000000_0
2016-05-18 00:57:33,430 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:33,430 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - DB9A5CB9962A4D25B5C198C1647842DC initialized RecordReader from event
2016-05-18 00:57:33,432 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:33,433 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: DB9A5CB9962A4D25B5C198C1647842DC
2016-05-18 00:57:33,433 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 4B1C2C92F1654F429347E06A2FB09C62
2016-05-18 00:57:33,433 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 4B1C2C92F1654F429347E06A2FB09C62
2016-05-18 00:57:33,435 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:33,438 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:33,439 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D83293AA11544C1397049AC88FD8F2DF using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@3aed5f9c, comparator=cascading.tuple.hadoop.util.TupleComparator@7548a5cd, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:33,440 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D83293AA11544C1397049AC88FD8F2DF: , UsingHashComparator=false
2016-05-18 00:57:33,468 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:33,468 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:33,468 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D83293AA11544C1397049AC88FD8F2DF: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:33,469 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D83293AA11544C1397049AC88FD8F2DF: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:33,469 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:33,469 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B10EC62DB61A4427B4FA58319CB707C6, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:33,472 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533053201_0001_1_00_000000_0
2016-05-18 00:57:33,473 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B10EC62DB61A4427B4FA58319CB707C6, mem on close (mb), free: 216, total: 357, max: 672
2016-05-18 00:57:33,473 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D83293AA11544C1397049AC88FD8F2DF: Starting flush of map output
2016-05-18 00:57:33,473 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D83293AA11544C1397049AC88FD8F2DF: Span0.length = 20, perItem = 6
2016-05-18 00:57:33,474 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D83293AA11544C1397049AC88FD8F2DF: done sorting span=0, length=20, time=1
2016-05-18 00:57:33,474 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D83293AA11544C1397049AC88FD8F2DF: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:33,475 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D83293AA11544C1397049AC88FD8F2DF: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001_wd/localmode-local-dir/output/attempt_1463533053201_0001_1_00_000000_0_10093_0/file.out
2016-05-18 00:57:33,479 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:33 Completed running task attempt: attempt_1463533053201_0001_1_00_000000_0
2016-05-18 00:57:33,480 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533053201_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:33,480 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533053201_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:33,480 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533053201_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533053469, Process_Duration=3, Process_End_Time=1463533053472, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:33,482 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533053201_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:33,482 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:33,482 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:33,482 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:33,482 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533053201_0001_00_000001
2016-05-18 00:57:33,482 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B10EC62DB61A4427B4FA58319CB707C6, taskAttemptId=attempt_1463533053201_0001_1_00_000000_0, creationTime=1463533053396, allocationTime=1463533053401, startTime=1463533053403, finishTime=1463533053482, timeTaken=79, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:33,483 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533053201_0001_1_00_000000_0
2016-05-18 00:57:33,484 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:TASK_FINISHED]: vertexName=B10EC62DB61A4427B4FA58319CB707C6, taskId=task_1463533053201_0001_1_00_000000, startTime=1463533053403, finishTime=1463533053484, timeTaken=81, status=SUCCEEDED, successfulAttemptID=attempt_1463533053201_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533053469, Process_Duration=3, Process_End_Time=1463533053472, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:33,484 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:33,485 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,486 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:VERTEX_FINISHED]: vertexName=B10EC62DB61A4427B4FA58319CB707C6, vertexId=vertex_1463533053201_0001_1_00, initRequestedTime=1463533053382, initedTime=1463533053393, startRequestedTime=1463533053384, startedTime=1463533053393, finishTime=1463533053485, timeTaken=92, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533053469, Process_Duration=3, Process_End_Time=1463533053472, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533053403, firstTasksToStart=[ task_1463533053201_0001_1_00_000000 ], lastTaskFinishTime=1463533053482, lastTasksToFinish=[ task_1463533053201_0001_1_00_000000 ], minTaskDuration=79, maxTaskDuration=79, avgTaskDuration=79.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533053201_0001_1_00_000000 ], longestDurationTasks=[ task_1463533053201_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:33,486 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:33,486 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533053201_0001_00_000001, stoppedTime=1463533053486, exitStatus=0
2016-05-18 00:57:33,486 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF] attempt: attempt_1463533053201_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:33,486 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:33,486 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D83293AA11544C1397049AC88FD8F2DF
2016-05-18 00:57:33,486 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B10EC62DB61A4427B4FA58319CB707C6, destinationVertexName=D83293AA11544C1397049AC88FD8F2DF. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:33,487 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]
2016-05-18 00:57:33,486 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:33,487 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:TASK_STARTED]: vertexName=D83293AA11544C1397049AC88FD8F2DF, taskId=task_1463533053201_0001_1_01_000000, scheduledTime=1463533053487, launchTime=1463533053487
2016-05-18 00:57:33,488 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533053201_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:33,488 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]
2016-05-18 00:57:33,490 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533053201_0001_00_000002, launchTime=1463533053490
2016-05-18 00:57:33,491 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533053201_0001_00_000002
2016-05-18 00:57:33,492 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533053201_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533053201_0001_1_01_000000_0
2016-05-18 00:57:33,492 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533053201_0001_1_01_000000_0] started. Is using containerId: [container_1463533053201_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:33 Starting to run new task attempt: attempt_1463533053201_0001_1_01_000000_0
2016-05-18 00:57:33,492 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533053201_0001_1_01_000000_0
2016-05-18 00:57:33,493 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:33,493 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533053201_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:33,493 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533053201_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:33,493 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:33,493 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533053201_0001_00_000001, Exiting
2016-05-18 00:57:33,494 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533053201_0001_00_000001
2016-05-18 00:57:33,494 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D83293AA11544C1397049AC88FD8F2DF, taskAttemptId=attempt_1463533053201_0001_1_01_000000_0, startTime=1463533053491, containerId=container_1463533053201_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:33,494 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533053201_0001_00_000001
2016-05-18 00:57:33,494 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C] every/(1/1) ...semblyplatform/-1_ARGS_-1, VertexName: D83293AA11544C1397049AC88FD8F2DF, VertexParallelism: 1, TaskAttemptID:attempt_1463533053201_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B10EC62DB61A4427B4FA58319CB707C6, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=881A5ED590074B72991F324054E4D823, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:33,495 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:33,496 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533053201_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:33,496 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533053201_0001_1_01_000000_0
2016-05-18 00:57:33,496 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533053201_0001_00_000001 completed successfully
2016-05-18 00:57:33,497 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533053201_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]
2016-05-18 00:57:33,497 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533053201_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:33,499 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:33,501 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:33,510 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D83293AA11544C1397049AC88FD8F2DF, ordinal: 1
2016-05-18 00:57:33,510 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D83293AA11544C1397049AC88FD8F2DF, mem on start (mb), free: 211, total: 357, max: 672
2016-05-18 00:57:33,510 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:33,510 INFO output.MROutput (MROutput.java:initialize(421)) - 881A5ED590074B72991F324054E4D823: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:33,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:33,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:33,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:33,511 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B10EC62DB61A4427B4FA58319CB707C6:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [881A5ED590074B72991F324054E4D823:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:33,512 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:33,512 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:33,512 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:33,512 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B10EC62DB61A4427B4FA58319CB707C6:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [881A5ED590074B72991F324054E4D823:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:33,513 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:33,514 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B10EC62DB61A4427B4FA58319CB707C6: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:33,514 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B10EC62DB61A4427B4FA58319CB707C6: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:33,515 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B10EC62DB61A4427B4FA58319CB707C6 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:33,516 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B10EC62DB61A4427B4FA58319CB707C6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:33,517 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B10EC62DB61A4427B4FA58319CB707C6
2016-05-18 00:57:33,517 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:33,517 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:33,517 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:33,518 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533053201_0001_1_01_000000_0
2016-05-18 00:57:33,520 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B10EC62DB61A4427B4FA58319CB707C6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:33,521 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533053201_0001_1_00_000000_0_10093} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533053521, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:33,521 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:33,521 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B10EC62DB61A4427B4FA58319CB707C6
2016-05-18 00:57:33,522 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 4B1C2C92F1654F429347E06A2FB09C62
2016-05-18 00:57:33,523 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:33,523 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_-1"], id: 881A5ED590074B72991F324054E4D823
2016-05-18 00:57:33,523 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B10EC62DB61A4427B4FA58319CB707C6, wasInterrupted=false
2016-05-18 00:57:33,523 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_-1"]
2016-05-18 00:57:33,525 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:33,525 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 4B1C2C92F1654F429347E06A2FB09C62, for 1 inputs
2016-05-18 00:57:33,526 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:33,528 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:33,528 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001_wd/localmode-local-dir/output/attempt_1463533053201_0001_1_00_000000_0_10093/file.out, len=181, isLocal=true
2016-05-18 00:57:33,529 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:33,529 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:33,529 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:33,530 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:33,530 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B10EC62DB61A4427B4FA58319CB707C6
2016-05-18 00:57:33,531 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D83293AA11544C1397049AC88FD8F2DF, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:33,531 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B10EC62DB61A4427B4FA58319CB707C6: Shuffle Runner thread complete
2016-05-18 00:57:33,532 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:33,532 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B10EC62DB61A4427B4FA58319CB707C6: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:33,539 INFO output.MROutput (MROutput.java:close(526)) - 881A5ED590074B72991F324054E4D823 closed
2016-05-18 00:57:33,540 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533053201_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:33,540 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533053201_0001_1_01_000000_0
2016-05-18 00:57:33,540 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D83293AA11544C1397049AC88FD8F2DF, mem on close (mb), free: 208, total: 357, max: 672
2016-05-18 00:57:33,540 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B10EC62DB61A4427B4FA58319CB707C6
2016-05-18 00:57:33,541 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B10EC62DB61A4427B4FA58319CB707C6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:33,541 INFO output.MROutput (MROutput.java:close(526)) - 881A5ED590074B72991F324054E4D823 closed
2016-05-18 00:57:33,541 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:33 Completed running task attempt: attempt_1463533053201_0001_1_01_000000_0
2016-05-18 00:57:33,541 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533053201_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:33,541 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533053201_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:33,541 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533053201_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533053531, Process_Duration=9, Process_End_Time=1463533053540, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:33,543 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533053201_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:33,543 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:33,543 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:33,543 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:33,543 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533053201_0001_00_000002
2016-05-18 00:57:33,543 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D83293AA11544C1397049AC88FD8F2DF, taskAttemptId=attempt_1463533053201_0001_1_01_000000_0, creationTime=1463533053488, allocationTime=1463533053489, startTime=1463533053491, finishTime=1463533053543, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:33,544 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533053201_0001_1_01_000000_0
2016-05-18 00:57:33,544 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:TASK_FINISHED]: vertexName=D83293AA11544C1397049AC88FD8F2DF, taskId=task_1463533053201_0001_1_01_000000, startTime=1463533053491, finishTime=1463533053544, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463533053201_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533053531, Process_Duration=9, Process_End_Time=1463533053540, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:33,544 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:33,544 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF]
2016-05-18 00:57:33,545 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:VERTEX_FINISHED]: vertexName=D83293AA11544C1397049AC88FD8F2DF, vertexId=vertex_1463533053201_0001_1_01, initRequestedTime=1463533053384, initedTime=1463533053388, startRequestedTime=1463533053394, startedTime=1463533053394, finishTime=1463533053544, timeTaken=150, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533053531, Process_Duration=9, Process_End_Time=1463533053540, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533053491, firstTasksToStart=[ task_1463533053201_0001_1_01_000000 ], lastTaskFinishTime=1463533053543, lastTasksToFinish=[ task_1463533053201_0001_1_01_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533053201_0001_1_01_000000 ], longestDurationTasks=[ task_1463533053201_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:33,545 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:33,545 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:33,545 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:33,545 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533053201_0001_1_00 [B10EC62DB61A4427B4FA58319CB707C6]
2016-05-18 00:57:33,545 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533053201_0001_1
2016-05-18 00:57:33,545 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533053201_0001_1
2016-05-18 00:57:33,546 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533053201_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:33,546 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 881A5ED590074B72991F324054E4D823 for vertex: vertex_1463533053201_0001_1_01 [D83293AA11544C1397049AC88FD8F2DF], outputName: 881A5ED590074B72991F324054E4D823
2016-05-18 00:57:33,546 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533053201_0001_00_000002, stoppedTime=1463533053546, exitStatus=0
2016-05-18 00:57:33,546 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533053201_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:33,553 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:881A5ED590074B72991F324054E4D823 of vertex/vertexGroup:D83293AA11544C1397049AC88FD8F2DF isVertexGroupOutput:false
2016-05-18 00:57:33,553 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:33,554 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533053201_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:33,554 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533053201_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:33,558 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533053201_0001_00_000002, Exiting
2016-05-18 00:57:33,558 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533053201_0001_00_000002
2016-05-18 00:57:33,558 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533053201_0001_00_000002
2016-05-18 00:57:33,555 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533053201_0001_1, queueSize=0
2016-05-18 00:57:33,558 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533053201_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:33,559 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533053201_0001_00_000002 completed successfully
2016-05-18 00:57:33,559 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053201_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533053201_0001_1, startTime=1463533053381, finishTime=1463533053553, timeTaken=172, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=360, AM_GC_TIME_MILLIS=14, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066107000, Process_Duration=12, Process_End_Time=2927066107012, Read_Duration=2, Tuples_Read=40, Tuples_Written=40, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:33,559 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533053201_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:33,560 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533053201_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:33 Completed Dag: dag_1463533053201_0001_1
2016-05-18 00:57:33,560 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533053201_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:33,560 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533053201_0001_1_post
2016-05-18 00:57:33,560 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:33,560 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:33,560 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:33,561 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C] every/(1/1) ...semblyplatform/-1_ARGS_-1, with id=dag_1463533053201_0001_1
2016-05-18 00:57:33,561 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C] every/(1/1) ...semblyplatform/-1_ARGS_-1, with id=dag_1463533053201_0001_1
2016-05-18 00:57:33,568 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...semblyplatform/-1_ARGS_-1, applicationId=application_1463533053201_0001
2016-05-18 00:57:33,570 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:33,570 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:33,571 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:33,571 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:33,571 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:33,571 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:33,572 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:33,573 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:33,573 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:33,574 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:33,574 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:33,574 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:33,574 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:33,574 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:33,578 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/7DB833BCEDC14ACCA357FFF4409B99C6/EC987070A70A4738B6B2486F3486332C/.tez/application_1463533053201_0001
2016-05-18 00:57:33,578 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_-1/_temporary
2016-05-18 00:57:33,579 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.404
2016-05-18 00:57:33,650 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_ARGS_-1
2016-05-18 00:57:33,674 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:33,674 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:33,703 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.003
2016-05-18 00:57:33,708 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.008
2016-05-18 00:57:33,708 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:33,709 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:33,709 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_ARGS_'field'
2016-05-18 00:57:33,732 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:33,733 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:33,771 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:33,782 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.024
2016-05-18 00:57:33,782 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:33,783 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:33,783 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:33,806 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:33,816 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_RESULTS
2016-05-18 00:57:33,866 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:33,866 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:33,867 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_RESULTS"]
2016-05-18 00:57:33,867 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:33,867 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:33,867 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:33,868 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...yplatform/-1_ARGS_RESULTS
2016-05-18 00:57:33,869 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:33,870 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:33,871 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:33,881 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:33,886 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:33,887 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:33,887 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:33,897 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001 doesn't exist and is created
2016-05-18 00:57:33,912 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:33,912 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:33,913 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001_wd
2016-05-18 00:57:33,922 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533053887_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:33,923 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:33,923 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:33,924 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:33,924 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:33,925 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:33,925 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:33,925 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:33,927 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:33,927 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:33,927 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:33,927 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:33,927 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:33,927 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:33,927 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:33,927 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:33,927 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:33,928 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:33,928 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:33,928 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:33,930 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:33,931 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:33,931 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:33,931 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001_wd/localmode-log-dir
2016-05-18 00:57:33,932 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001_wd/localmode-log-dir/history.txt.appattempt_1463533053887_0001_000000, maxErrors=10
2016-05-18 00:57:33,932 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:33,932 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533053887_0001_000000, appSubmitTime=1463533053922, launchTime=1463533053922
2016-05-18 00:57:33,935 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:33,940 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:49199
2016-05-18 00:57:33,940 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533053887
2016-05-18 00:57:33,942 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533053887_0001_000000, startTime=1463533053942
2016-05-18 00:57:33,942 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:34,012 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:34,013 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:34,013 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...yplatform/-1_ARGS_RESULTS, applicationId=application_1463533053887_0001, dagName=[57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8] every/(1/1) ...yplatform/-1_ARGS_RESULTS
2016-05-18 00:57:34,043 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8] every/(1/1) ...yplatform/-1_ARGS_RESULTS
2016-05-18 00:57:34,044 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:34,044 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533053887_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001_wd/localmode-log-dir/dag_1463533053887_0001_1.dot
2016-05-18 00:57:34,045 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001_wd/localmode-log-dir/dag_1463533053887_0001_1-tez-dag.pb.txt
2016-05-18 00:57:34,051 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533053887_0001_1
2016-05-18 00:57:34,051 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,051 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8] every/(1/1) ...yplatform/-1_ARGS_RESULTS
2016-05-18 00:57:34 Running Dag: dag_1463533053887_0001_1
2016-05-18 00:57:34,060 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533053887_0001_1, submitTime=1463533054044
2016-05-18 00:57:34,061 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001/tez-conf.pb" } size: 36913 timestamp: 1463533053000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:34,061 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,062 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:34,062 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001/tez-conf.pb" } size: 36913 timestamp: 1463533053000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:34,062 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]
2016-05-18 00:57:34,063 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:34,063 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:34,063 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533053887_0001_1, initTime=1463533054063
2016-05-18 00:57:34,064 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533053887_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:34,064 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:34,064 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:DAG_STARTED]: dagID=dag_1463533053887_0001_1, startTime=1463533054064
2016-05-18 00:57:34,064 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533053887_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:34,064 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: DD9444D4BB274862993879EC3809EE7F : {92673F995A7C412396EE237136062A07={InputName=92673F995A7C412396EE237136062A07}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:34,064 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 92673F995A7C412396EE237136062A07, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:34,064 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,064 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,065 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,065 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,065 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:34,065 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]
2016-05-18 00:57:34,066 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...yplatform/-1_ARGS_RESULTS, applicationId=application_1463533053887_0001, dagName=[57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8] every/(1/1) ...yplatform/-1_ARGS_RESULTS
2016-05-18 00:57:34,067 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 92673F995A7C412396EE237136062A07 on vertex vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,069 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:34,069 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]
2016-05-18 00:57:34,070 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]
2016-05-18 00:57:34,069 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 92673F995A7C412396EE237136062A07 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:34,072 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533053887_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:34,072 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9], numAdditionalOutputs=1
2016-05-18 00:57:34,072 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=02013A2E9A37433BA4427405DC1B3030, vertex=vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:34,073 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:34,073 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:34,074 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:34,074 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E497441F67AC4D889D660A5F25AB42A9:02013A2E9A37433BA4427405DC1B3030 using oldmapred API
2016-05-18 00:57:34,074 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=02013A2E9A37433BA4427405DC1B3030, vertexName=E497441F67AC4D889D660A5F25AB42A9, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:34,074 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:34,076 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 92673F995A7C412396EE237136062A07 on vertex vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,076 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E497441F67AC4D889D660A5F25AB42A9, vertexId=vertex_1463533053887_0001_1_01, initRequestedTime=1463533054065, initedTime=1463533054072, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:34,076 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:34,077 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:34,077 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {92673F995A7C412396EE237136062A07=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:34,078 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] parallelism set to 1
2016-05-18 00:57:34,078 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:34,078 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DD9444D4BB274862993879EC3809EE7F, vertexId=vertex_1463533053887_0001_1_00, initRequestedTime=1463533054064, initedTime=1463533054078, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:34,079 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] with distanceFromRoot: 0
2016-05-18 00:57:34,079 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:34,079 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533053887_0001_1_00, startRequestedTime=1463533054065, startedTime=1463533054079
2016-05-18 00:57:34,079 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533053887_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:34,079 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:34,079 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533053887_0001_1_00 for vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9] numStartedSources: 1 numSources: 1
2016-05-18 00:57:34,079 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9] with distanceFromRoot: 1
2016-05-18 00:57:34,080 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533053887_0001_1_01, startRequestedTime=1463533054079, startedTime=1463533054080
2016-05-18 00:57:34,080 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:34,083 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E497441F67AC4D889D660A5F25AB42A9 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:34,084 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: DD9444D4BB274862993879EC3809EE7F in vertex: E497441F67AC4D889D660A5F25AB42A9 numBipartiteSourceTasks: 1
2016-05-18 00:57:34,084 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in DD9444D4BB274862993879EC3809EE7F
2016-05-18 00:57:34,084 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,085 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:TASK_STARTED]: vertexName=DD9444D4BB274862993879EC3809EE7F, taskId=task_1463533053887_0001_1_00_000000, scheduledTime=1463533054085, launchTime=1463533054085
2016-05-18 00:57:34,085 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,086 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:34,088 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533053887_0001_00_000001, launchTime=1463533054087
2016-05-18 00:57:34,088 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533053887_0001_00_000001
2016-05-18 00:57:34,089 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533053887_0001_1_00_000000_0] started. Is using containerId: [container_1463533053887_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:34,089 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DD9444D4BB274862993879EC3809EE7F, taskAttemptId=attempt_1463533053887_0001_1_00_000000_0, startTime=1463533054089, containerId=container_1463533053887_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:34,090 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533053887_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533053887_0001_1_00_000000_0
2016-05-18 00:57:34 Starting to run new task attempt: attempt_1463533053887_0001_1_00_000000_0
2016-05-18 00:57:34,090 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533053887_0001_1_00_000000_0
2016-05-18 00:57:34,091 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,091 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:34,091 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8] every/(1/1) ...yplatform/-1_ARGS_RESULTS, VertexName: DD9444D4BB274862993879EC3809EE7F, VertexParallelism: 1, TaskAttemptID:attempt_1463533053887_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=92673F995A7C412396EE237136062A07, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E497441F67AC4D889D660A5F25AB42A9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:34,091 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:34,092 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533053887_0001_1_00_000000_0
2016-05-18 00:57:34,092 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533053887_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:34,092 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533053887_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:34,093 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533053887_0001, with dag id: dag_1463533053887_0001_1
2016-05-18 00:57:34,097 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:34,098 INFO input.MRInput (MRInput.java:initialize(440)) - 92673F995A7C412396EE237136062A07 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:34,100 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 92673F995A7C412396EE237136062A07
2016-05-18 00:57:34,104 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DD9444D4BB274862993879EC3809EE7F, ordinal: 0
2016-05-18 00:57:34,104 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DD9444D4BB274862993879EC3809EE7F, mem on start (mb), free: 300, total: 363, max: 672
2016-05-18 00:57:34,104 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:34,104 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:34,105 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:34,105 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:34,105 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[92673F995A7C412396EE237136062A07:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E497441F67AC4D889D660A5F25AB42A9:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:34,105 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:34,105 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:34,106 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:34,106 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[92673F995A7C412396EE237136062A07:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E497441F67AC4D889D660A5F25AB42A9:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:34,106 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:34,106 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 92673F995A7C412396EE237136062A07 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:34,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:34,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:34,106 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 92673F995A7C412396EE237136062A07
2016-05-18 00:57:34,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:34,108 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533053887_0001_1_00_000000_0
2016-05-18 00:57:34,109 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:34,109 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 92673F995A7C412396EE237136062A07 initialized RecordReader from event
2016-05-18 00:57:34,110 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:34,111 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 92673F995A7C412396EE237136062A07
2016-05-18 00:57:34,111 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 5BC8ECE3DB794F64828E2D25E28C32F7
2016-05-18 00:57:34,111 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 5BC8ECE3DB794F64828E2D25E28C32F7
2016-05-18 00:57:34,112 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:34,115 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:34,115 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E497441F67AC4D889D660A5F25AB42A9 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6f4cdeaf, comparator=cascading.tuple.hadoop.util.TupleComparator@56bf2c59, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:34,115 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E497441F67AC4D889D660A5F25AB42A9: , UsingHashComparator=false
2016-05-18 00:57:34,128 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:34,128 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:34,128 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E497441F67AC4D889D660A5F25AB42A9: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:34,128 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E497441F67AC4D889D660A5F25AB42A9: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:34,129 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:34,129 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DD9444D4BB274862993879EC3809EE7F, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:34,132 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533053887_0001_1_00_000000_0
2016-05-18 00:57:34,133 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DD9444D4BB274862993879EC3809EE7F, mem on close (mb), free: 198, total: 363, max: 672
2016-05-18 00:57:34,133 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E497441F67AC4D889D660A5F25AB42A9: Starting flush of map output
2016-05-18 00:57:34,133 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E497441F67AC4D889D660A5F25AB42A9: Span0.length = 20, perItem = 6
2016-05-18 00:57:34,134 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E497441F67AC4D889D660A5F25AB42A9: done sorting span=0, length=20, time=1
2016-05-18 00:57:34,134 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E497441F67AC4D889D660A5F25AB42A9: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:34,135 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E497441F67AC4D889D660A5F25AB42A9: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001_wd/localmode-local-dir/output/attempt_1463533053887_0001_1_00_000000_0_10099_0/file.out
2016-05-18 00:57:34,140 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:34 Completed running task attempt: attempt_1463533053887_0001_1_00_000000_0
2016-05-18 00:57:34,141 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533053887_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:34,141 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533053887_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:34,142 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533053887_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533054129, Process_Duration=3, Process_End_Time=1463533054132, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:34,144 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533053887_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:34,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DD9444D4BB274862993879EC3809EE7F, taskAttemptId=attempt_1463533053887_0001_1_00_000000_0, creationTime=1463533054085, allocationTime=1463533054087, startTime=1463533054089, finishTime=1463533054144, timeTaken=55, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:34,144 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:34,144 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:34,145 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,145 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533053887_0001_1_00_000000_0
2016-05-18 00:57:34,145 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533053887_0001_00_000001
2016-05-18 00:57:34,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:TASK_FINISHED]: vertexName=DD9444D4BB274862993879EC3809EE7F, taskId=task_1463533053887_0001_1_00_000000, startTime=1463533054089, finishTime=1463533054145, timeTaken=56, status=SUCCEEDED, successfulAttemptID=attempt_1463533053887_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533054129, Process_Duration=3, Process_End_Time=1463533054132, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:34,146 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:34,146 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,147 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:VERTEX_FINISHED]: vertexName=DD9444D4BB274862993879EC3809EE7F, vertexId=vertex_1463533053887_0001_1_00, initRequestedTime=1463533054064, initedTime=1463533054078, startRequestedTime=1463533054065, startedTime=1463533054079, finishTime=1463533054146, timeTaken=67, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533054129, Process_Duration=3, Process_End_Time=1463533054132, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533054089, firstTasksToStart=[ task_1463533053887_0001_1_00_000000 ], lastTaskFinishTime=1463533054144, lastTasksToFinish=[ task_1463533053887_0001_1_00_000000 ], minTaskDuration=55, maxTaskDuration=55, avgTaskDuration=55.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533053887_0001_1_00_000000 ], longestDurationTasks=[ task_1463533053887_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:34,147 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:34,147 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9] attempt: attempt_1463533053887_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:34,148 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533053887_0001_00_000001, stoppedTime=1463533054148, exitStatus=0
2016-05-18 00:57:34,148 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E497441F67AC4D889D660A5F25AB42A9
2016-05-18 00:57:34,148 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=DD9444D4BB274862993879EC3809EE7F, destinationVertexName=E497441F67AC4D889D660A5F25AB42A9. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:34,148 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]
2016-05-18 00:57:34,148 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:34,149 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:34,149 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533053887_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:34,149 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:TASK_STARTED]: vertexName=E497441F67AC4D889D660A5F25AB42A9, taskId=task_1463533053887_0001_1_01_000000, scheduledTime=1463533054149, launchTime=1463533054149
2016-05-18 00:57:34,149 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]
2016-05-18 00:57:34,151 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533053887_0001_00_000002, launchTime=1463533054151
2016-05-18 00:57:34,151 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533053887_0001_00_000002
2016-05-18 00:57:34,152 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533053887_0001_1_01_000000_0] started. Is using containerId: [container_1463533053887_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:34,152 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E497441F67AC4D889D660A5F25AB42A9, taskAttemptId=attempt_1463533053887_0001_1_01_000000_0, startTime=1463533054152, containerId=container_1463533053887_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:34,152 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533053887_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533053887_0001_1_01_000000_0
2016-05-18 00:57:34 Starting to run new task attempt: attempt_1463533053887_0001_1_01_000000_0
2016-05-18 00:57:34,153 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533053887_0001_1_01_000000_0
2016-05-18 00:57:34,153 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,153 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:34,154 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8] every/(1/1) ...yplatform/-1_ARGS_RESULTS, VertexName: E497441F67AC4D889D660A5F25AB42A9, VertexParallelism: 1, TaskAttemptID:attempt_1463533053887_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DD9444D4BB274862993879EC3809EE7F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=02013A2E9A37433BA4427405DC1B3030, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:34,154 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:34,154 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533053887_0001_1_01_000000_0
2016-05-18 00:57:34,156 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533053887_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:34,156 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533053887_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:34,157 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:34,158 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:34,163 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533053887_0001_00_000001, Exiting
2016-05-18 00:57:34,163 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533053887_0001_00_000001
2016-05-18 00:57:34,163 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533053887_0001_00_000001
2016-05-18 00:57:34,164 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533053887_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]
2016-05-18 00:57:34,164 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533053887_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:34,164 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533053887_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:34,164 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533053887_0001_00_000001 completed successfully
2016-05-18 00:57:34,165 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E497441F67AC4D889D660A5F25AB42A9, ordinal: 1
2016-05-18 00:57:34,165 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E497441F67AC4D889D660A5F25AB42A9, mem on start (mb), free: 193, total: 363, max: 672
2016-05-18 00:57:34,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:34,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:34,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:34,167 INFO output.MROutput (MROutput.java:initialize(421)) - 02013A2E9A37433BA4427405DC1B3030: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:34,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:34,167 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[02013A2E9A37433BA4427405DC1B3030:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [DD9444D4BB274862993879EC3809EE7F:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:34,168 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:34,168 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:34,168 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:34,168 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[02013A2E9A37433BA4427405DC1B3030:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [DD9444D4BB274862993879EC3809EE7F:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:34,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:34,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DD9444D4BB274862993879EC3809EE7F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:34,169 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:34,169 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:34,169 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - DD9444D4BB274862993879EC3809EE7F: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:34,169 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - DD9444D4BB274862993879EC3809EE7F: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:34,170 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: DD9444D4BB274862993879EC3809EE7F with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:34,174 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DD9444D4BB274862993879EC3809EE7F
2016-05-18 00:57:34,174 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:34,175 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533053887_0001_1_01_000000_0
2016-05-18 00:57:34,176 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DD9444D4BB274862993879EC3809EE7F: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:34,177 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533053887_0001_1_00_000000_0_10099} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533054177, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:34,178 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : DD9444D4BB274862993879EC3809EE7F
2016-05-18 00:57:34,178 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:34,178 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: DD9444D4BB274862993879EC3809EE7F, wasInterrupted=false
2016-05-18 00:57:34,178 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:34,180 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 5BC8ECE3DB794F64828E2D25E28C32F7
2016-05-18 00:57:34,180 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_RESULTS"], id: 02013A2E9A37433BA4427405DC1B3030
2016-05-18 00:57:34,180 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:34,181 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_RESULTS"]
2016-05-18 00:57:34,181 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:34,181 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 5BC8ECE3DB794F64828E2D25E28C32F7, for 1 inputs
2016-05-18 00:57:34,182 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:34,182 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001_wd/localmode-local-dir/output/attempt_1463533053887_0001_1_00_000000_0_10099/file.out, len=181, isLocal=true
2016-05-18 00:57:34,182 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:34,182 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:34,182 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:34,183 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:34,183 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : DD9444D4BB274862993879EC3809EE7F
2016-05-18 00:57:34,183 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - DD9444D4BB274862993879EC3809EE7F: Shuffle Runner thread complete
2016-05-18 00:57:34,183 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E497441F67AC4D889D660A5F25AB42A9, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:57:34,184 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:34,184 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - DD9444D4BB274862993879EC3809EE7F: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:34,190 INFO output.MROutput (MROutput.java:close(526)) - 02013A2E9A37433BA4427405DC1B3030 closed
2016-05-18 00:57:34,190 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533053887_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:34,191 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533053887_0001_1_01_000000_0
2016-05-18 00:57:34,191 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E497441F67AC4D889D660A5F25AB42A9, mem on close (mb), free: 190, total: 363, max: 672
2016-05-18 00:57:34,191 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: DD9444D4BB274862993879EC3809EE7F
2016-05-18 00:57:34,191 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DD9444D4BB274862993879EC3809EE7F: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:34,191 INFO output.MROutput (MROutput.java:close(526)) - 02013A2E9A37433BA4427405DC1B3030 closed
2016-05-18 00:57:34,191 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:34 Completed running task attempt: attempt_1463533053887_0001_1_01_000000_0
2016-05-18 00:57:34,192 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533053887_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:34,192 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533053887_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:34,192 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533053887_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533054183, Process_Duration=8, Process_End_Time=1463533054191, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:34,193 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533053887_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:34,193 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E497441F67AC4D889D660A5F25AB42A9, taskAttemptId=attempt_1463533053887_0001_1_01_000000_0, creationTime=1463533054149, allocationTime=1463533054150, startTime=1463533054152, finishTime=1463533054193, timeTaken=41, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:34,194 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533053887_0001_1_01_000000_0
2016-05-18 00:57:34,193 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:34,194 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:34,194 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:TASK_FINISHED]: vertexName=E497441F67AC4D889D660A5F25AB42A9, taskId=task_1463533053887_0001_1_01_000000, startTime=1463533054152, finishTime=1463533054194, timeTaken=42, status=SUCCEEDED, successfulAttemptID=attempt_1463533053887_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533054183, Process_Duration=8, Process_End_Time=1463533054191, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:34,195 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:34,195 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9]
2016-05-18 00:57:34,195 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,196 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:VERTEX_FINISHED]: vertexName=E497441F67AC4D889D660A5F25AB42A9, vertexId=vertex_1463533053887_0001_1_01, initRequestedTime=1463533054065, initedTime=1463533054072, startRequestedTime=1463533054079, startedTime=1463533054080, finishTime=1463533054195, timeTaken=115, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533054183, Process_Duration=8, Process_End_Time=1463533054191, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533054152, firstTasksToStart=[ task_1463533053887_0001_1_01_000000 ], lastTaskFinishTime=1463533054193, lastTasksToFinish=[ task_1463533053887_0001_1_01_000000 ], minTaskDuration=41, maxTaskDuration=41, avgTaskDuration=41.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533053887_0001_1_01_000000 ], longestDurationTasks=[ task_1463533053887_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:34,196 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:34,196 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533053887_0001_00_000002, stoppedTime=1463533054196, exitStatus=0
2016-05-18 00:57:34,196 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:34,196 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:34,196 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533053887_0001_1_00 [DD9444D4BB274862993879EC3809EE7F]
2016-05-18 00:57:34,196 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533053887_0001_1
2016-05-18 00:57:34,197 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533053887_0001_1
2016-05-18 00:57:34,197 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533053887_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:34,197 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 02013A2E9A37433BA4427405DC1B3030 for vertex: vertex_1463533053887_0001_1_01 [E497441F67AC4D889D660A5F25AB42A9], outputName: 02013A2E9A37433BA4427405DC1B3030
2016-05-18 00:57:34,197 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533053887_0001_00_000002
2016-05-18 00:57:34,197 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533053887_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:34,197 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533053887_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:34,197 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533053887_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:34,197 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533053887_0001_00_000002, Exiting
2016-05-18 00:57:34,197 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533053887_0001_00_000002
2016-05-18 00:57:34,197 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533053887_0001_00_000002
2016-05-18 00:57:34,198 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533053887_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:34,198 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533053887_0001_00_000002 completed successfully
2016-05-18 00:57:34,203 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:02013A2E9A37433BA4427405DC1B3030 of vertex/vertexGroup:E497441F67AC4D889D660A5F25AB42A9 isVertexGroupOutput:false
2016-05-18 00:57:34,203 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:34,205 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533053887_0001_1, queueSize=0
2016-05-18 00:57:34,206 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533053887_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533053887_0001_1, startTime=1463533054064, finishTime=1463533054204, timeTaken=140, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=240, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066108312, Process_Duration=11, Process_End_Time=2927066108323, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:34,206 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533053887_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:34,206 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533053887_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:34 Completed Dag: dag_1463533053887_0001_1
2016-05-18 00:57:34,206 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533053887_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:34,207 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533053887_0001_1_post
2016-05-18 00:57:34,207 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,207 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:34,207 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:34,207 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8] every/(1/1) ...yplatform/-1_ARGS_RESULTS, with id=dag_1463533053887_0001_1
2016-05-18 00:57:34,207 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8] every/(1/1) ...yplatform/-1_ARGS_RESULTS, with id=dag_1463533053887_0001_1
2016-05-18 00:57:34,216 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...yplatform/-1_ARGS_RESULTS, applicationId=application_1463533053887_0001
2016-05-18 00:57:34,218 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:34,218 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:34,218 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:34,218 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:34,218 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:34,218 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:34,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:34,219 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:34,219 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:34,219 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:34,220 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:34,220 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:34,220 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:34,222 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:34,226 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/57CDECCE40EB40659064F7992417ADCB/F151BEC6B94A4B6EB20A7599B11D5DF8/.tez/application_1463533053887_0001
2016-05-18 00:57:34,227 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_RESULTS/_temporary
2016-05-18 00:57:34,227 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.361
2016-05-18 00:57:34,297 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_ARGS_RESULTS
2016-05-18 00:57:34,319 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:34,319 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:34,352 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.007
2016-05-18 00:57:34,353 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.008
2016-05-18 00:57:34,353 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:34,353 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:34,354 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_ARGS_ALL
2016-05-18 00:57:34,377 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:34,378 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:34,412 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.009
2016-05-18 00:57:34,425 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.022
2016-05-18 00:57:34,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:34,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:34,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:34,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:34,448 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_REPLACE
2016-05-18 00:57:34,497 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:34,497 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:34,528 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:34,528 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:34,528 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_REPLACE"]
2016-05-18 00:57:34,529 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:34,529 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:34,529 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:34,530 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...yplatform/-1_ARGS_REPLACE
2016-05-18 00:57:34,530 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:34,543 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:34,548 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:34,548 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:34,548 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:34,559 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001 doesn't exist and is created
2016-05-18 00:57:34,575 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:34,575 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:34,576 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001_wd
2016-05-18 00:57:34,589 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533054548_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:34,589 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:34,589 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:34,591 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:34,592 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:34,592 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:34,592 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:34,593 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:34,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:34,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:34,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:34,594 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:34,595 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:34,595 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:34,595 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:34,595 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:34,595 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:34,596 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:34,596 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:34,596 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:34,598 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:34,599 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:34,599 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:34,599 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001_wd/localmode-log-dir
2016-05-18 00:57:34,599 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001_wd/localmode-log-dir/history.txt.appattempt_1463533054548_0001_000000, maxErrors=10
2016-05-18 00:57:34,600 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:34,600 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533054548_0001_000000, appSubmitTime=1463533054588, launchTime=1463533054589
2016-05-18 00:57:34,604 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:34,609 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:42716
2016-05-18 00:57:34,609 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533054548
2016-05-18 00:57:34,611 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533054548_0001_000000, startTime=1463533054611
2016-05-18 00:57:34,611 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:34,675 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:34,675 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:34,676 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...yplatform/-1_ARGS_REPLACE, applicationId=application_1463533054548_0001, dagName=[8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69] every/(1/1) ...yplatform/-1_ARGS_REPLACE
2016-05-18 00:57:34,692 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69] every/(1/1) ...yplatform/-1_ARGS_REPLACE
2016-05-18 00:57:34,692 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:34,693 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533054548_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001_wd/localmode-log-dir/dag_1463533054548_0001_1.dot
2016-05-18 00:57:34,693 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001_wd/localmode-log-dir/dag_1463533054548_0001_1-tez-dag.pb.txt
2016-05-18 00:57:34,700 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533054548_0001_1
2016-05-18 00:57:34,700 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,700 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69] every/(1/1) ...yplatform/-1_ARGS_REPLACE
2016-05-18 00:57:34 Running Dag: dag_1463533054548_0001_1
2016-05-18 00:57:34,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533054548_0001_1, submitTime=1463533054692
2016-05-18 00:57:34,710 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001/tez-conf.pb" } size: 36913 timestamp: 1463533054000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:34,710 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,711 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:34,711 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001/tez-conf.pb" } size: 36913 timestamp: 1463533054000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:34,712 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]
2016-05-18 00:57:34,712 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:34,712 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:34,713 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533054548_0001_1, initTime=1463533054712
2016-05-18 00:57:34,713 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533054548_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:34,714 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:34,714 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:DAG_STARTED]: dagID=dag_1463533054548_0001_1, startTime=1463533054714
2016-05-18 00:57:34,714 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533054548_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:34,714 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C9C458A0D64E4865AEB38530172C00D7 : {CCC2FDEABC5E4E01A88CA399A6377F4F={InputName=CCC2FDEABC5E4E01A88CA399A6377F4F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:34,714 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: CCC2FDEABC5E4E01A88CA399A6377F4F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:34,714 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,714 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...yplatform/-1_ARGS_REPLACE, applicationId=application_1463533054548_0001, dagName=[8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69] every/(1/1) ...yplatform/-1_ARGS_REPLACE
2016-05-18 00:57:34,714 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,715 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,715 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,715 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:34,715 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]
2016-05-18 00:57:34,716 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: CCC2FDEABC5E4E01A88CA399A6377F4F on vertex vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,719 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input CCC2FDEABC5E4E01A88CA399A6377F4F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:34,719 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:34,719 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]
2016-05-18 00:57:34,719 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]
2016-05-18 00:57:34,720 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533054548_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:34,721 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5], numAdditionalOutputs=1
2016-05-18 00:57:34,721 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=F1CBB9553F3F4F53ABB1D5EF34B6964C, vertex=vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:34,722 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:34,722 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:34,722 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:34,723 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:34,723 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for AC752E4B28DB4F348100352198510DF5:F1CBB9553F3F4F53ABB1D5EF34B6964C using oldmapred API
2016-05-18 00:57:34,723 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=F1CBB9553F3F4F53ABB1D5EF34B6964C, vertexName=AC752E4B28DB4F348100352198510DF5, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:34,724 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: CCC2FDEABC5E4E01A88CA399A6377F4F on vertex vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,725 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AC752E4B28DB4F348100352198510DF5, vertexId=vertex_1463533054548_0001_1_01, initRequestedTime=1463533054715, initedTime=1463533054721, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:34,726 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:34,727 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:34,727 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {CCC2FDEABC5E4E01A88CA399A6377F4F=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:34,727 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] parallelism set to 1
2016-05-18 00:57:34,728 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:34,728 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C9C458A0D64E4865AEB38530172C00D7, vertexId=vertex_1463533054548_0001_1_00, initRequestedTime=1463533054714, initedTime=1463533054728, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:34,729 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] with distanceFromRoot: 0
2016-05-18 00:57:34,729 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:34,729 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533054548_0001_1_00, startRequestedTime=1463533054715, startedTime=1463533054729
2016-05-18 00:57:34,730 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533054548_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:34,731 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:34,731 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533054548_0001_1_00 for vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5] numStartedSources: 1 numSources: 1
2016-05-18 00:57:34,731 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5] with distanceFromRoot: 1
2016-05-18 00:57:34,731 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C9C458A0D64E4865AEB38530172C00D7
2016-05-18 00:57:34,732 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,733 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533054548_0001_1_01, startRequestedTime=1463533054731, startedTime=1463533054732
2016-05-18 00:57:34,733 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:34,733 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533054548_0001, with dag id: dag_1463533054548_0001_1
2016-05-18 00:57:34,734 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:TASK_STARTED]: vertexName=C9C458A0D64E4865AEB38530172C00D7, taskId=task_1463533054548_0001_1_00_000000, scheduledTime=1463533054734, launchTime=1463533054734
2016-05-18 00:57:34,734 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: AC752E4B28DB4F348100352198510DF5 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:34,735 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C9C458A0D64E4865AEB38530172C00D7 in vertex: AC752E4B28DB4F348100352198510DF5 numBipartiteSourceTasks: 1
2016-05-18 00:57:34,735 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,736 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:34,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533054548_0001_00_000001, launchTime=1463533054739
2016-05-18 00:57:34,740 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533054548_0001_00_000001
2016-05-18 00:57:34,741 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533054548_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533054548_0001_1_00_000000_0
2016-05-18 00:57:34,741 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533054548_0001_1_00_000000_0] started. Is using containerId: [container_1463533054548_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:34,741 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C9C458A0D64E4865AEB38530172C00D7, taskAttemptId=attempt_1463533054548_0001_1_00_000000_0, startTime=1463533054741, containerId=container_1463533054548_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:34 Starting to run new task attempt: attempt_1463533054548_0001_1_00_000000_0
2016-05-18 00:57:34,741 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533054548_0001_1_00_000000_0
2016-05-18 00:57:34,742 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,742 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:34,742 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69] every/(1/1) ...yplatform/-1_ARGS_REPLACE, VertexName: C9C458A0D64E4865AEB38530172C00D7, VertexParallelism: 1, TaskAttemptID:attempt_1463533054548_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CCC2FDEABC5E4E01A88CA399A6377F4F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=AC752E4B28DB4F348100352198510DF5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:34,742 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:34,743 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533054548_0001_1_00_000000_0
2016-05-18 00:57:34,743 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533054548_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:34,743 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533054548_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:34,746 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:34,747 INFO input.MRInput (MRInput.java:initialize(440)) - CCC2FDEABC5E4E01A88CA399A6377F4F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:34,747 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: CCC2FDEABC5E4E01A88CA399A6377F4F
2016-05-18 00:57:34,752 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C9C458A0D64E4865AEB38530172C00D7, ordinal: 0
2016-05-18 00:57:34,753 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C9C458A0D64E4865AEB38530172C00D7, mem on start (mb), free: 262, total: 351, max: 672
2016-05-18 00:57:34,753 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:34,753 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:34,753 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:34,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:34,754 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CCC2FDEABC5E4E01A88CA399A6377F4F:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [AC752E4B28DB4F348100352198510DF5:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:34,754 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:34,755 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:34,755 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:34,755 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CCC2FDEABC5E4E01A88CA399A6377F4F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [AC752E4B28DB4F348100352198510DF5:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:34,755 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:34,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CCC2FDEABC5E4E01A88CA399A6377F4F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:34,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CCC2FDEABC5E4E01A88CA399A6377F4F
2016-05-18 00:57:34,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:34,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:34,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:34,757 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533054548_0001_1_00_000000_0
2016-05-18 00:57:34,758 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:34,758 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - CCC2FDEABC5E4E01A88CA399A6377F4F initialized RecordReader from event
2016-05-18 00:57:34,760 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:34,760 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: CCC2FDEABC5E4E01A88CA399A6377F4F
2016-05-18 00:57:34,760 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: F8278371BEF44BA3805B3469973620BB
2016-05-18 00:57:34,760 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] F8278371BEF44BA3805B3469973620BB
2016-05-18 00:57:34,762 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:34,765 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:34,765 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - AC752E4B28DB4F348100352198510DF5 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@50987248, comparator=cascading.tuple.hadoop.util.TupleComparator@7c31d80c, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:34,765 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for AC752E4B28DB4F348100352198510DF5: , UsingHashComparator=false
2016-05-18 00:57:34,778 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:34,779 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:34,779 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for AC752E4B28DB4F348100352198510DF5: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:34,779 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - AC752E4B28DB4F348100352198510DF5: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:34,780 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:34,780 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C9C458A0D64E4865AEB38530172C00D7, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:34,783 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533054548_0001_1_00_000000_0
2016-05-18 00:57:34,784 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C9C458A0D64E4865AEB38530172C00D7, mem on close (mb), free: 160, total: 351, max: 672
2016-05-18 00:57:34,784 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - AC752E4B28DB4F348100352198510DF5: Starting flush of map output
2016-05-18 00:57:34,784 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - AC752E4B28DB4F348100352198510DF5: Span0.length = 20, perItem = 6
2016-05-18 00:57:34,785 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - AC752E4B28DB4F348100352198510DF5: done sorting span=0, length=20, time=1
2016-05-18 00:57:34,785 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - AC752E4B28DB4F348100352198510DF5: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:34,786 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - AC752E4B28DB4F348100352198510DF5: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001_wd/localmode-local-dir/output/attempt_1463533054548_0001_1_00_000000_0_10104_0/file.out
2016-05-18 00:57:34,790 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:34 Completed running task attempt: attempt_1463533054548_0001_1_00_000000_0
2016-05-18 00:57:34,791 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533054548_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:34,791 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533054548_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:34,792 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533054548_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533054781, Process_Duration=2, Process_End_Time=1463533054783, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:34,794 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533054548_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:34,794 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:34,794 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:34,794 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C9C458A0D64E4865AEB38530172C00D7, taskAttemptId=attempt_1463533054548_0001_1_00_000000_0, creationTime=1463533054734, allocationTime=1463533054738, startTime=1463533054741, finishTime=1463533054794, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:34,794 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,795 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533054548_0001_1_00_000000_0
2016-05-18 00:57:34,796 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:TASK_FINISHED]: vertexName=C9C458A0D64E4865AEB38530172C00D7, taskId=task_1463533054548_0001_1_00_000000, startTime=1463533054741, finishTime=1463533054795, timeTaken=54, status=SUCCEEDED, successfulAttemptID=attempt_1463533054548_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533054781, Process_Duration=2, Process_End_Time=1463533054783, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:34,796 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:34,796 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,796 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533054548_0001_00_000001
2016-05-18 00:57:34,797 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:VERTEX_FINISHED]: vertexName=C9C458A0D64E4865AEB38530172C00D7, vertexId=vertex_1463533054548_0001_1_00, initRequestedTime=1463533054714, initedTime=1463533054728, startRequestedTime=1463533054715, startedTime=1463533054729, finishTime=1463533054796, timeTaken=67, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533054781, Process_Duration=2, Process_End_Time=1463533054783, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533054741, firstTasksToStart=[ task_1463533054548_0001_1_00_000000 ], lastTaskFinishTime=1463533054794, lastTasksToFinish=[ task_1463533054548_0001_1_00_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533054548_0001_1_00_000000 ], longestDurationTasks=[ task_1463533054548_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:34,797 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:34,797 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533054548_0001_00_000001, stoppedTime=1463533054797, exitStatus=0
2016-05-18 00:57:34,797 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5] attempt: attempt_1463533054548_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:34,798 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:34,798 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AC752E4B28DB4F348100352198510DF5
2016-05-18 00:57:34,798 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:34,798 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C9C458A0D64E4865AEB38530172C00D7, destinationVertexName=AC752E4B28DB4F348100352198510DF5. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:34,798 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]
2016-05-18 00:57:34,798 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533054548_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:34,799 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:TASK_STARTED]: vertexName=AC752E4B28DB4F348100352198510DF5, taskId=task_1463533054548_0001_1_01_000000, scheduledTime=1463533054799, launchTime=1463533054799
2016-05-18 00:57:34,799 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]
2016-05-18 00:57:34,801 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533054548_0001_00_000002, launchTime=1463533054801
2016-05-18 00:57:34,801 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533054548_0001_00_000002
2016-05-18 00:57:34,802 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533054548_0001_1_01_000000_0] started. Is using containerId: [container_1463533054548_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:34,802 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AC752E4B28DB4F348100352198510DF5, taskAttemptId=attempt_1463533054548_0001_1_01_000000_0, startTime=1463533054802, containerId=container_1463533054548_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:34,803 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533054548_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533054548_0001_1_01_000000_0
2016-05-18 00:57:34 Starting to run new task attempt: attempt_1463533054548_0001_1_01_000000_0
2016-05-18 00:57:34,803 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533054548_0001_1_01_000000_0
2016-05-18 00:57:34,803 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,804 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:34,804 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69] every/(1/1) ...yplatform/-1_ARGS_REPLACE, VertexName: AC752E4B28DB4F348100352198510DF5, VertexParallelism: 1, TaskAttemptID:attempt_1463533054548_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C9C458A0D64E4865AEB38530172C00D7, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=F1CBB9553F3F4F53ABB1D5EF34B6964C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:34,804 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:34,805 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533054548_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]
2016-05-18 00:57:34,806 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533054548_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:34,806 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533054548_0001_1_01_000000_0
2016-05-18 00:57:34,807 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533054548_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:34,807 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533054548_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:34,807 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533054548_0001_00_000001, Exiting
2016-05-18 00:57:34,807 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533054548_0001_00_000001
2016-05-18 00:57:34,807 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533054548_0001_00_000001
2016-05-18 00:57:34,809 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:34,811 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533054548_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:34,811 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533054548_0001_00_000001 completed successfully
2016-05-18 00:57:34,812 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:34,817 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AC752E4B28DB4F348100352198510DF5, ordinal: 1
2016-05-18 00:57:34,817 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AC752E4B28DB4F348100352198510DF5, mem on start (mb), free: 156, total: 351, max: 672
2016-05-18 00:57:34,818 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:34,818 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:34,818 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:34,824 INFO output.MROutput (MROutput.java:initialize(421)) - F1CBB9553F3F4F53ABB1D5EF34B6964C: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:34,824 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:34,824 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[C9C458A0D64E4865AEB38530172C00D7:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [F1CBB9553F3F4F53ABB1D5EF34B6964C:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:34,825 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:34,825 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:34,825 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:34,825 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[C9C458A0D64E4865AEB38530172C00D7:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [F1CBB9553F3F4F53ABB1D5EF34B6964C:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:34,825 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:34,825 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C9C458A0D64E4865AEB38530172C00D7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:34,826 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:34,826 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:34,826 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C9C458A0D64E4865AEB38530172C00D7: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:34,826 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C9C458A0D64E4865AEB38530172C00D7: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:34,827 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C9C458A0D64E4865AEB38530172C00D7 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:34,828 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C9C458A0D64E4865AEB38530172C00D7
2016-05-18 00:57:34,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:34,830 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533054548_0001_1_01_000000_0
2016-05-18 00:57:34,832 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C9C458A0D64E4865AEB38530172C00D7: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:34,834 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:34,835 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: F8278371BEF44BA3805B3469973620BB
2016-05-18 00:57:34,835 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_REPLACE"], id: F1CBB9553F3F4F53ABB1D5EF34B6964C
2016-05-18 00:57:34,836 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_REPLACE"]
2016-05-18 00:57:34,836 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] F8278371BEF44BA3805B3469973620BB, for 1 inputs
2016-05-18 00:57:34,838 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533054548_0001_1_00_000000_0_10104} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533054838, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:34,838 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C9C458A0D64E4865AEB38530172C00D7
2016-05-18 00:57:34,839 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:34,839 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C9C458A0D64E4865AEB38530172C00D7, wasInterrupted=false
2016-05-18 00:57:34,840 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:34,841 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:34,842 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:34,842 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001_wd/localmode-local-dir/output/attempt_1463533054548_0001_1_00_000000_0_10104/file.out, len=181, isLocal=true
2016-05-18 00:57:34,842 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:34,842 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:34,842 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:34,842 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:34,842 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C9C458A0D64E4865AEB38530172C00D7
2016-05-18 00:57:34,842 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C9C458A0D64E4865AEB38530172C00D7: Shuffle Runner thread complete
2016-05-18 00:57:34,843 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AC752E4B28DB4F348100352198510DF5, all 1 inputs ready in: 00:00:00.006
2016-05-18 00:57:34,844 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:34,844 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C9C458A0D64E4865AEB38530172C00D7: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:34,852 INFO output.MROutput (MROutput.java:close(526)) - F1CBB9553F3F4F53ABB1D5EF34B6964C closed
2016-05-18 00:57:34,853 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533054548_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:34,854 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533054548_0001_1_01_000000_0
2016-05-18 00:57:34,854 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AC752E4B28DB4F348100352198510DF5, mem on close (mb), free: 153, total: 351, max: 672
2016-05-18 00:57:34,855 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C9C458A0D64E4865AEB38530172C00D7
2016-05-18 00:57:34,855 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C9C458A0D64E4865AEB38530172C00D7: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:34,855 INFO output.MROutput (MROutput.java:close(526)) - F1CBB9553F3F4F53ABB1D5EF34B6964C closed
2016-05-18 00:57:34,856 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:34 Completed running task attempt: attempt_1463533054548_0001_1_01_000000_0
2016-05-18 00:57:34,857 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533054548_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:34,857 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533054548_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:34,857 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533054548_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533054843, Process_Duration=11, Process_End_Time=1463533054854, Read_Duration=4, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:34,859 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533054548_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:34,859 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:34,859 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AC752E4B28DB4F348100352198510DF5, taskAttemptId=attempt_1463533054548_0001_1_01_000000_0, creationTime=1463533054799, allocationTime=1463533054800, startTime=1463533054802, finishTime=1463533054859, timeTaken=57, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:34,859 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533054548_0001_1_01_000000_0
2016-05-18 00:57:34,860 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:TASK_FINISHED]: vertexName=AC752E4B28DB4F348100352198510DF5, taskId=task_1463533054548_0001_1_01_000000, startTime=1463533054802, finishTime=1463533054859, timeTaken=57, status=SUCCEEDED, successfulAttemptID=attempt_1463533054548_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533054843, Process_Duration=11, Process_End_Time=1463533054854, Read_Duration=4, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:34,860 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:34,860 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5]
2016-05-18 00:57:34,859 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:34,861 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,861 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:VERTEX_FINISHED]: vertexName=AC752E4B28DB4F348100352198510DF5, vertexId=vertex_1463533054548_0001_1_01, initRequestedTime=1463533054715, initedTime=1463533054721, startRequestedTime=1463533054731, startedTime=1463533054732, finishTime=1463533054860, timeTaken=128, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533054843, Process_Duration=11, Process_End_Time=1463533054854, Read_Duration=4, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533054802, firstTasksToStart=[ task_1463533054548_0001_1_01_000000 ], lastTaskFinishTime=1463533054859, lastTasksToFinish=[ task_1463533054548_0001_1_01_000000 ], minTaskDuration=57, maxTaskDuration=57, avgTaskDuration=57.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533054548_0001_1_01_000000 ], longestDurationTasks=[ task_1463533054548_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:34,861 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:34,862 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533054548_0001_00_000002
2016-05-18 00:57:34,862 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533054548_0001_00_000002, stoppedTime=1463533054862, exitStatus=0
2016-05-18 00:57:34,862 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533054548_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:34,862 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:34,862 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:34,863 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533054548_0001_1_00 [C9C458A0D64E4865AEB38530172C00D7]
2016-05-18 00:57:34,863 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533054548_0001_1
2016-05-18 00:57:34,862 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533054548_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:34,863 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533054548_0001_1
2016-05-18 00:57:34,863 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: F1CBB9553F3F4F53ABB1D5EF34B6964C for vertex: vertex_1463533054548_0001_1_01 [AC752E4B28DB4F348100352198510DF5], outputName: F1CBB9553F3F4F53ABB1D5EF34B6964C
2016-05-18 00:57:34,863 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533054548_0001_00_000002, Exiting
2016-05-18 00:57:34,864 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533054548_0001_00_000002
2016-05-18 00:57:34,864 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533054548_0001_00_000002
2016-05-18 00:57:34,864 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533054548_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:34,864 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533054548_0001_00_000002 completed successfully
2016-05-18 00:57:34,865 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533054548_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:34,865 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533054548_0001_00_000002
2016-05-18 00:57:34,870 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:F1CBB9553F3F4F53ABB1D5EF34B6964C of vertex/vertexGroup:AC752E4B28DB4F348100352198510DF5 isVertexGroupOutput:false
2016-05-18 00:57:34,870 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:34,873 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533054548_0001_1, queueSize=0
2016-05-18 00:57:34,873 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533054548_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533054548_0001_1, startTime=1463533054713, finishTime=1463533054871, timeTaken=158, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=300, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066109624, Process_Duration=13, Process_End_Time=2927066109637, Read_Duration=6, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:34,874 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533054548_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:34,874 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533054548_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:34 Completed Dag: dag_1463533054548_0001_1
2016-05-18 00:57:34,874 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533054548_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:34,874 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533054548_0001_1_post
2016-05-18 00:57:34,874 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:34,874 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:34,874 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:34,874 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69] every/(1/1) ...yplatform/-1_ARGS_REPLACE, with id=dag_1463533054548_0001_1
2016-05-18 00:57:34,875 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69] every/(1/1) ...yplatform/-1_ARGS_REPLACE, with id=dag_1463533054548_0001_1
2016-05-18 00:57:34,882 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...yplatform/-1_ARGS_REPLACE, applicationId=application_1463533054548_0001
2016-05-18 00:57:34,884 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:34,884 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:34,884 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:34,884 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:34,884 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:34,885 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:34,885 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:34,885 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:34,886 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:34,886 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:34,886 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:34,886 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:34,886 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:34,887 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:34,891 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8D7E1C31D70B430785002565A17B3BAF/EEE04A82F3D343D2A6CCBD3D9512DF69/.tez/application_1463533054548_0001
2016-05-18 00:57:34,891 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_REPLACE/_temporary
2016-05-18 00:57:34,892 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.363
2016-05-18 00:57:34,962 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_ARGS_REPLACE
2016-05-18 00:57:34,986 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:34,986 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:35,025 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 00:57:35,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 00:57:35,031 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:35,031 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:35,031 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:35,049 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:35,059 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_SWAP
2016-05-18 00:57:35,120 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:35,120 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:35,134 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:35,135 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:35,135 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_SWAP"]
2016-05-18 00:57:35,135 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:35,136 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:35,137 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:35,137 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...mblyplatform/-1_ARGS_SWAP
2016-05-18 00:57:35,138 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:35,150 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:35,154 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:35,154 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:35,154 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:35,163 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001 doesn't exist and is created
2016-05-18 00:57:35,177 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:35,178 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:35,179 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001_wd
2016-05-18 00:57:35,191 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533055154_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:35,192 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:35,192 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:35,194 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:35,194 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:35,195 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:35,195 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:35,196 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:35,198 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:35,198 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:35,198 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:35,198 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:35,199 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:35,199 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:35,199 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:35,199 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:35,199 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:35,199 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:35,200 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:35,200 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:35,202 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:35,203 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:35,203 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:35,204 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001_wd/localmode-log-dir
2016-05-18 00:57:35,204 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001_wd/localmode-log-dir/history.txt.appattempt_1463533055154_0001_000000, maxErrors=10
2016-05-18 00:57:35,204 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:35,205 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533055154_0001_000000, appSubmitTime=1463533055191, launchTime=1463533055191
2016-05-18 00:57:35,207 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:35,210 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:48733
2016-05-18 00:57:35,210 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533055154
2016-05-18 00:57:35,213 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533055154_0001_000000, startTime=1463533055213
2016-05-18 00:57:35,213 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:35,278 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:35,278 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:35,278 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...mblyplatform/-1_ARGS_SWAP, applicationId=application_1463533055154_0001, dagName=[F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655] every/(1/1) ...mblyplatform/-1_ARGS_SWAP
2016-05-18 00:57:35,294 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655] every/(1/1) ...mblyplatform/-1_ARGS_SWAP
2016-05-18 00:57:35,295 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:35,296 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533055154_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001_wd/localmode-log-dir/dag_1463533055154_0001_1.dot
2016-05-18 00:57:35,297 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001_wd/localmode-log-dir/dag_1463533055154_0001_1-tez-dag.pb.txt
2016-05-18 00:57:35,304 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533055154_0001_1
2016-05-18 00:57:35,304 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,304 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655] every/(1/1) ...mblyplatform/-1_ARGS_SWAP
2016-05-18 00:57:35 Running Dag: dag_1463533055154_0001_1
2016-05-18 00:57:35,314 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533055154_0001_1, submitTime=1463533055294
2016-05-18 00:57:35,315 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001/tez-conf.pb" } size: 36913 timestamp: 1463533055000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:35,316 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,317 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:35,318 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001/tez-conf.pb" } size: 36913 timestamp: 1463533055000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:35,318 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]
2016-05-18 00:57:35,319 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:35,319 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:35,320 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533055154_0001_1, initTime=1463533055320
2016-05-18 00:57:35,321 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533055154_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:35,321 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:35,321 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:DAG_STARTED]: dagID=dag_1463533055154_0001_1, startTime=1463533055321
2016-05-18 00:57:35,321 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533055154_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:35,321 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...mblyplatform/-1_ARGS_SWAP, applicationId=application_1463533055154_0001, dagName=[F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655] every/(1/1) ...mblyplatform/-1_ARGS_SWAP
2016-05-18 00:57:35,322 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: DD0B54FEEDDF4FF39348139210D935AE : {6939104DA2A74C559967D7F30E9515FE={InputName=6939104DA2A74C559967D7F30E9515FE}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:35,322 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 6939104DA2A74C559967D7F30E9515FE, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:35,322 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,322 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,322 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,322 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,324 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:35,324 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]
2016-05-18 00:57:35,324 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 6939104DA2A74C559967D7F30E9515FE on vertex vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,326 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 6939104DA2A74C559967D7F30E9515FE asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:35,327 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:35,327 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]
2016-05-18 00:57:35,327 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]
2016-05-18 00:57:35,328 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:35,328 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533055154_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:35,329 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0], numAdditionalOutputs=1
2016-05-18 00:57:35,329 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=33A78D7E77634CBAAC3C75EB9C736989, vertex=vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:35,328 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:35,331 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:35,331 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:35,332 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F3A28707B5A844C39994961F2C5BCAE0:33A78D7E77634CBAAC3C75EB9C736989 using oldmapred API
2016-05-18 00:57:35,332 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=33A78D7E77634CBAAC3C75EB9C736989, vertexName=F3A28707B5A844C39994961F2C5BCAE0, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:35,332 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 6939104DA2A74C559967D7F30E9515FE on vertex vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,333 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F3A28707B5A844C39994961F2C5BCAE0, vertexId=vertex_1463533055154_0001_1_01, initRequestedTime=1463533055324, initedTime=1463533055329, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:35,333 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:35,335 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:35,336 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {6939104DA2A74C559967D7F30E9515FE=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:35,336 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] parallelism set to 1
2016-05-18 00:57:35,336 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:35,337 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DD0B54FEEDDF4FF39348139210D935AE, vertexId=vertex_1463533055154_0001_1_00, initRequestedTime=1463533055322, initedTime=1463533055337, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:35,337 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] with distanceFromRoot: 0
2016-05-18 00:57:35,337 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:35,337 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533055154_0001_1_00, startRequestedTime=1463533055324, startedTime=1463533055337
2016-05-18 00:57:35,337 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533055154_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:35,338 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:35,338 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533055154_0001_1_00 for vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0] numStartedSources: 1 numSources: 1
2016-05-18 00:57:35,338 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0] with distanceFromRoot: 1
2016-05-18 00:57:35,338 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533055154_0001_1_01, startRequestedTime=1463533055338, startedTime=1463533055338
2016-05-18 00:57:35,339 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:35,340 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F3A28707B5A844C39994961F2C5BCAE0 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:35,340 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in DD0B54FEEDDF4FF39348139210D935AE
2016-05-18 00:57:35,340 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,341 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: DD0B54FEEDDF4FF39348139210D935AE in vertex: F3A28707B5A844C39994961F2C5BCAE0 numBipartiteSourceTasks: 1
2016-05-18 00:57:35,341 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:TASK_STARTED]: vertexName=DD0B54FEEDDF4FF39348139210D935AE, taskId=task_1463533055154_0001_1_00_000000, scheduledTime=1463533055341, launchTime=1463533055341
2016-05-18 00:57:35,341 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,343 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:35,344 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533055154_0001_00_000001, launchTime=1463533055344
2016-05-18 00:57:35,345 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533055154_0001_00_000001
2016-05-18 00:57:35,346 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533055154_0001_1_00_000000_0] started. Is using containerId: [container_1463533055154_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:35,346 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DD0B54FEEDDF4FF39348139210D935AE, taskAttemptId=attempt_1463533055154_0001_1_00_000000_0, startTime=1463533055346, containerId=container_1463533055154_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:35,346 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533055154_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533055154_0001_1_00_000000_0
2016-05-18 00:57:35 Starting to run new task attempt: attempt_1463533055154_0001_1_00_000000_0
2016-05-18 00:57:35,347 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533055154_0001_1_00_000000_0
2016-05-18 00:57:35,347 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,348 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:35,348 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655] every/(1/1) ...mblyplatform/-1_ARGS_SWAP, VertexName: DD0B54FEEDDF4FF39348139210D935AE, VertexParallelism: 1, TaskAttemptID:attempt_1463533055154_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=6939104DA2A74C559967D7F30E9515FE, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F3A28707B5A844C39994961F2C5BCAE0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:35,348 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:35,349 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533055154_0001, with dag id: dag_1463533055154_0001_1
2016-05-18 00:57:35,349 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533055154_0001_1_00_000000_0
2016-05-18 00:57:35,353 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:35,354 INFO input.MRInput (MRInput.java:initialize(440)) - 6939104DA2A74C559967D7F30E9515FE using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:35,354 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 6939104DA2A74C559967D7F30E9515FE
2016-05-18 00:57:35,356 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533055154_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:35,356 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533055154_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:35,359 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DD0B54FEEDDF4FF39348139210D935AE, ordinal: 0
2016-05-18 00:57:35,359 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DD0B54FEEDDF4FF39348139210D935AE, mem on start (mb), free: 263, total: 362, max: 672
2016-05-18 00:57:35,359 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:35,359 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:35,359 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:35,359 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:35,359 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6939104DA2A74C559967D7F30E9515FE:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F3A28707B5A844C39994961F2C5BCAE0:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:35,359 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:35,360 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:35,360 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:35,360 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6939104DA2A74C559967D7F30E9515FE:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F3A28707B5A844C39994961F2C5BCAE0:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:35,360 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:35,360 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 6939104DA2A74C559967D7F30E9515FE being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:35,360 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:35,360 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 6939104DA2A74C559967D7F30E9515FE
2016-05-18 00:57:35,360 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:35,361 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:35,361 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533055154_0001_1_00_000000_0
2016-05-18 00:57:35,363 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:35,364 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 6939104DA2A74C559967D7F30E9515FE
2016-05-18 00:57:35,364 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 9874843634D34C0AB1B10AE03D451B04
2016-05-18 00:57:35,364 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 9874843634D34C0AB1B10AE03D451B04
2016-05-18 00:57:35,364 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:35,365 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 6939104DA2A74C559967D7F30E9515FE initialized RecordReader from event
2016-05-18 00:57:35,365 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:35,367 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:35,367 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F3A28707B5A844C39994961F2C5BCAE0 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4894a870, comparator=cascading.tuple.hadoop.util.TupleComparator@17be764a, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:35,368 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F3A28707B5A844C39994961F2C5BCAE0: , UsingHashComparator=false
2016-05-18 00:57:35,380 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:35,380 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:35,380 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F3A28707B5A844C39994961F2C5BCAE0: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:35,381 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F3A28707B5A844C39994961F2C5BCAE0: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:35,381 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:35,381 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DD0B54FEEDDF4FF39348139210D935AE, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:35,387 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533055154_0001_1_00_000000_0
2016-05-18 00:57:35,387 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DD0B54FEEDDF4FF39348139210D935AE, mem on close (mb), free: 161, total: 362, max: 672
2016-05-18 00:57:35,387 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F3A28707B5A844C39994961F2C5BCAE0: Starting flush of map output
2016-05-18 00:57:35,387 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F3A28707B5A844C39994961F2C5BCAE0: Span0.length = 20, perItem = 6
2016-05-18 00:57:35,387 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F3A28707B5A844C39994961F2C5BCAE0: done sorting span=0, length=20, time=0
2016-05-18 00:57:35,387 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F3A28707B5A844C39994961F2C5BCAE0: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:35,389 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F3A28707B5A844C39994961F2C5BCAE0: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001_wd/localmode-local-dir/output/attempt_1463533055154_0001_1_00_000000_0_10110_0/file.out
2016-05-18 00:57:35,393 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:35 Completed running task attempt: attempt_1463533055154_0001_1_00_000000_0
2016-05-18 00:57:35,393 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533055154_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:35,393 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533055154_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:35,394 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533055154_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533055381, Process_Duration=6, Process_End_Time=1463533055387, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:35,395 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533055154_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:35,395 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:35,395 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DD0B54FEEDDF4FF39348139210D935AE, taskAttemptId=attempt_1463533055154_0001_1_00_000000_0, creationTime=1463533055341, allocationTime=1463533055343, startTime=1463533055346, finishTime=1463533055395, timeTaken=49, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:35,395 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:35,395 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,395 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533055154_0001_1_00_000000_0
2016-05-18 00:57:35,396 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533055154_0001_00_000001
2016-05-18 00:57:35,397 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:TASK_FINISHED]: vertexName=DD0B54FEEDDF4FF39348139210D935AE, taskId=task_1463533055154_0001_1_00_000000, startTime=1463533055346, finishTime=1463533055395, timeTaken=49, status=SUCCEEDED, successfulAttemptID=attempt_1463533055154_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533055381, Process_Duration=6, Process_End_Time=1463533055387, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:35,397 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:35,397 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,398 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:VERTEX_FINISHED]: vertexName=DD0B54FEEDDF4FF39348139210D935AE, vertexId=vertex_1463533055154_0001_1_00, initRequestedTime=1463533055322, initedTime=1463533055337, startRequestedTime=1463533055324, startedTime=1463533055337, finishTime=1463533055397, timeTaken=60, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533055381, Process_Duration=6, Process_End_Time=1463533055387, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533055346, firstTasksToStart=[ task_1463533055154_0001_1_00_000000 ], lastTaskFinishTime=1463533055395, lastTasksToFinish=[ task_1463533055154_0001_1_00_000000 ], minTaskDuration=49, maxTaskDuration=49, avgTaskDuration=49.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533055154_0001_1_00_000000 ], longestDurationTasks=[ task_1463533055154_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:35,399 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:35,399 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533055154_0001_00_000001, stoppedTime=1463533055399, exitStatus=0
2016-05-18 00:57:35,399 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0] attempt: attempt_1463533055154_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:35,399 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:35,399 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F3A28707B5A844C39994961F2C5BCAE0
2016-05-18 00:57:35,400 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=DD0B54FEEDDF4FF39348139210D935AE, destinationVertexName=F3A28707B5A844C39994961F2C5BCAE0. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:35,400 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]
2016-05-18 00:57:35,399 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:35,400 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533055154_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:35,400 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:TASK_STARTED]: vertexName=F3A28707B5A844C39994961F2C5BCAE0, taskId=task_1463533055154_0001_1_01_000000, scheduledTime=1463533055400, launchTime=1463533055400
2016-05-18 00:57:35,401 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]
2016-05-18 00:57:35,402 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533055154_0001_00_000002, launchTime=1463533055402
2016-05-18 00:57:35,403 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533055154_0001_00_000002
2016-05-18 00:57:35,404 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533055154_0001_1_01_000000_0] started. Is using containerId: [container_1463533055154_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:35,404 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F3A28707B5A844C39994961F2C5BCAE0, taskAttemptId=attempt_1463533055154_0001_1_01_000000_0, startTime=1463533055404, containerId=container_1463533055154_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:35,405 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533055154_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533055154_0001_1_01_000000_0
2016-05-18 00:57:35 Starting to run new task attempt: attempt_1463533055154_0001_1_01_000000_0
2016-05-18 00:57:35,405 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533055154_0001_1_01_000000_0
2016-05-18 00:57:35,406 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533055154_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:35,406 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533055154_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:35,406 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,406 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533055154_0001_00_000001, Exiting
2016-05-18 00:57:35,407 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:35,407 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533055154_0001_00_000001
2016-05-18 00:57:35,407 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655] every/(1/1) ...mblyplatform/-1_ARGS_SWAP, VertexName: F3A28707B5A844C39994961F2C5BCAE0, VertexParallelism: 1, TaskAttemptID:attempt_1463533055154_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DD0B54FEEDDF4FF39348139210D935AE, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=33A78D7E77634CBAAC3C75EB9C736989, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:35,407 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533055154_0001_00_000001
2016-05-18 00:57:35,408 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:35,408 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533055154_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:35,409 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533055154_0001_1_01_000000_0
2016-05-18 00:57:35,409 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533055154_0001_00_000001 completed successfully
2016-05-18 00:57:35,410 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533055154_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]
2016-05-18 00:57:35,410 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533055154_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:35,413 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:35,417 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:35,422 INFO output.MROutput (MROutput.java:initialize(421)) - 33A78D7E77634CBAAC3C75EB9C736989: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:35,424 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F3A28707B5A844C39994961F2C5BCAE0, ordinal: 1
2016-05-18 00:57:35,424 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F3A28707B5A844C39994961F2C5BCAE0, mem on start (mb), free: 155, total: 362, max: 672
2016-05-18 00:57:35,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:35,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:35,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:35,425 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:35,425 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DD0B54FEEDDF4FF39348139210D935AE:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [33A78D7E77634CBAAC3C75EB9C736989:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:35,425 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:35,425 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:35,426 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:35,426 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DD0B54FEEDDF4FF39348139210D935AE:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [33A78D7E77634CBAAC3C75EB9C736989:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:35,426 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:35,426 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DD0B54FEEDDF4FF39348139210D935AE being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:35,427 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - DD0B54FEEDDF4FF39348139210D935AE: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:35,427 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - DD0B54FEEDDF4FF39348139210D935AE: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:35,428 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: DD0B54FEEDDF4FF39348139210D935AE with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:35,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:35,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:35,430 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DD0B54FEEDDF4FF39348139210D935AE
2016-05-18 00:57:35,430 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:35,430 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533055154_0001_1_01_000000_0
2016-05-18 00:57:35,434 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DD0B54FEEDDF4FF39348139210D935AE: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:35,434 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:35,434 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 9874843634D34C0AB1B10AE03D451B04
2016-05-18 00:57:35,435 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_SWAP"], id: 33A78D7E77634CBAAC3C75EB9C736989
2016-05-18 00:57:35,435 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_SWAP"]
2016-05-18 00:57:35,435 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 9874843634D34C0AB1B10AE03D451B04, for 1 inputs
2016-05-18 00:57:35,436 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533055154_0001_1_00_000000_0_10110} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533055436, TimeTaken=2, Rate=0.09 MB/s
2016-05-18 00:57:35,436 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : DD0B54FEEDDF4FF39348139210D935AE
2016-05-18 00:57:35,436 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:35,437 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: DD0B54FEEDDF4FF39348139210D935AE, wasInterrupted=false
2016-05-18 00:57:35,437 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:35,438 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:35,439 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:35,439 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001_wd/localmode-local-dir/output/attempt_1463533055154_0001_1_00_000000_0_10110/file.out, len=181, isLocal=true
2016-05-18 00:57:35,439 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:35,439 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:35,439 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:35,440 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:35,440 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : DD0B54FEEDDF4FF39348139210D935AE
2016-05-18 00:57:35,440 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - DD0B54FEEDDF4FF39348139210D935AE: Shuffle Runner thread complete
2016-05-18 00:57:35,440 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F3A28707B5A844C39994961F2C5BCAE0, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:35,441 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:35,441 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - DD0B54FEEDDF4FF39348139210D935AE: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:35,446 INFO output.MROutput (MROutput.java:close(526)) - 33A78D7E77634CBAAC3C75EB9C736989 closed
2016-05-18 00:57:35,446 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533055154_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:35,446 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533055154_0001_1_01_000000_0
2016-05-18 00:57:35,447 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F3A28707B5A844C39994961F2C5BCAE0, mem on close (mb), free: 152, total: 362, max: 672
2016-05-18 00:57:35,447 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: DD0B54FEEDDF4FF39348139210D935AE
2016-05-18 00:57:35,447 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DD0B54FEEDDF4FF39348139210D935AE: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:35,447 INFO output.MROutput (MROutput.java:close(526)) - 33A78D7E77634CBAAC3C75EB9C736989 closed
2016-05-18 00:57:35,447 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:35 Completed running task attempt: attempt_1463533055154_0001_1_01_000000_0
2016-05-18 00:57:35,448 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533055154_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:35,448 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533055154_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:35,449 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533055154_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533055440, Process_Duration=6, Process_End_Time=1463533055446, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:35,450 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533055154_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:35,450 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F3A28707B5A844C39994961F2C5BCAE0, taskAttemptId=attempt_1463533055154_0001_1_01_000000_0, creationTime=1463533055400, allocationTime=1463533055401, startTime=1463533055404, finishTime=1463533055450, timeTaken=46, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:35,450 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:35,450 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:35,450 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533055154_0001_1_01_000000_0
2016-05-18 00:57:35,450 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:TASK_FINISHED]: vertexName=F3A28707B5A844C39994961F2C5BCAE0, taskId=task_1463533055154_0001_1_01_000000, startTime=1463533055404, finishTime=1463533055450, timeTaken=46, status=SUCCEEDED, successfulAttemptID=attempt_1463533055154_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533055440, Process_Duration=6, Process_End_Time=1463533055446, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:35,451 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:35,451 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0]
2016-05-18 00:57:35,451 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:VERTEX_FINISHED]: vertexName=F3A28707B5A844C39994961F2C5BCAE0, vertexId=vertex_1463533055154_0001_1_01, initRequestedTime=1463533055324, initedTime=1463533055329, startRequestedTime=1463533055338, startedTime=1463533055338, finishTime=1463533055451, timeTaken=113, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533055440, Process_Duration=6, Process_End_Time=1463533055446, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533055404, firstTasksToStart=[ task_1463533055154_0001_1_01_000000 ], lastTaskFinishTime=1463533055450, lastTasksToFinish=[ task_1463533055154_0001_1_01_000000 ], minTaskDuration=46, maxTaskDuration=46, avgTaskDuration=46.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533055154_0001_1_01_000000 ], longestDurationTasks=[ task_1463533055154_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:35,451 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:35,452 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:35,452 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:35,452 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533055154_0001_1_00 [DD0B54FEEDDF4FF39348139210D935AE]
2016-05-18 00:57:35,452 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533055154_0001_1
2016-05-18 00:57:35,452 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533055154_0001_1
2016-05-18 00:57:35,452 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533055154_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:35,452 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,452 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 33A78D7E77634CBAAC3C75EB9C736989 for vertex: vertex_1463533055154_0001_1_01 [F3A28707B5A844C39994961F2C5BCAE0], outputName: 33A78D7E77634CBAAC3C75EB9C736989
2016-05-18 00:57:35,453 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533055154_0001_00_000002, stoppedTime=1463533055453, exitStatus=0
2016-05-18 00:57:35,453 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533055154_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:35,453 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533055154_0001_00_000002
2016-05-18 00:57:35,454 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533055154_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:35,454 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533055154_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:35,454 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533055154_0001_00_000002, Exiting
2016-05-18 00:57:35,454 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533055154_0001_00_000002
2016-05-18 00:57:35,454 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533055154_0001_00_000002
2016-05-18 00:57:35,454 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533055154_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:35,454 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533055154_0001_00_000002 completed successfully
2016-05-18 00:57:35,459 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:33A78D7E77634CBAAC3C75EB9C736989 of vertex/vertexGroup:F3A28707B5A844C39994961F2C5BCAE0 isVertexGroupOutput:false
2016-05-18 00:57:35,459 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:35,461 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533055154_0001_1, queueSize=0
2016-05-18 00:57:35,461 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055154_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533055154_0001_1, startTime=1463533055321, finishTime=1463533055459, timeTaken=138, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=280, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066110821, Process_Duration=12, Process_End_Time=2927066110833, Read_Duration=4, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:35,461 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533055154_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:35,461 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533055154_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:35 Completed Dag: dag_1463533055154_0001_1
2016-05-18 00:57:35,462 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533055154_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:35,462 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533055154_0001_1_post
2016-05-18 00:57:35,462 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,462 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:35,462 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:35,462 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655] every/(1/1) ...mblyplatform/-1_ARGS_SWAP, with id=dag_1463533055154_0001_1
2016-05-18 00:57:35,462 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655] every/(1/1) ...mblyplatform/-1_ARGS_SWAP, with id=dag_1463533055154_0001_1
2016-05-18 00:57:35,470 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...mblyplatform/-1_ARGS_SWAP, applicationId=application_1463533055154_0001
2016-05-18 00:57:35,471 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:35,472 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:35,472 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:35,472 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:35,472 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:35,472 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:35,472 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:35,473 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:35,473 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:35,473 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:35,473 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:35,474 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:35,474 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:35,475 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:35,477 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F00E44FAF0A54E75B3E6C044F8244C2F/C3532071A2C7488FBC67A7A0547F6655/.tez/application_1463533055154_0001
2016-05-18 00:57:35,478 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/-1_ARGS_SWAP/_temporary
2016-05-18 00:57:35,478 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.344
2016-05-18 00:57:35,548 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//-1_ARGS_SWAP
2016-05-18 00:57:35,571 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:35,572 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:35,620 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 00:57:35,621 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:35,622 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:35,622 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:35,622 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:35,639 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:35,644 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_-1
2016-05-18 00:57:35,692 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:35,692 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:35,692 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_-1"]
2016-05-18 00:57:35,693 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:35,693 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:35,693 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:35,694 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...mblyplatform/0_'field'_-1
2016-05-18 00:57:35,694 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:35,706 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:35,710 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:35,711 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:35,712 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:35,721 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001 doesn't exist and is created
2016-05-18 00:57:35,732 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:35,733 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:35,736 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:35,736 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:35,737 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001_wd
2016-05-18 00:57:35,747 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533055711_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:35,748 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:35,748 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:35,750 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:35,750 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:35,750 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:35,750 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:35,751 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:35,753 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:35,753 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:35,753 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:35,754 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:35,754 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:35,754 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:35,755 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:35,755 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:35,755 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:35,755 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:35,756 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:35,756 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:35,758 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:35,758 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:35,759 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:35,759 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001_wd/localmode-log-dir
2016-05-18 00:57:35,759 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001_wd/localmode-log-dir/history.txt.appattempt_1463533055711_0001_000000, maxErrors=10
2016-05-18 00:57:35,759 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:35,760 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533055711_0001_000000, appSubmitTime=1463533055747, launchTime=1463533055747
2016-05-18 00:57:35,765 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:35,770 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:48239
2016-05-18 00:57:35,771 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533055711
2016-05-18 00:57:35,772 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533055711_0001_000000, startTime=1463533055772
2016-05-18 00:57:35,772 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:35,836 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:35,836 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:35,837 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...mblyplatform/0_'field'_-1, applicationId=application_1463533055711_0001, dagName=[9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720] every/(1/1) ...mblyplatform/0_'field'_-1
2016-05-18 00:57:35,853 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720] every/(1/1) ...mblyplatform/0_'field'_-1
2016-05-18 00:57:35,853 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:35,854 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533055711_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001_wd/localmode-log-dir/dag_1463533055711_0001_1.dot
2016-05-18 00:57:35,854 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001_wd/localmode-log-dir/dag_1463533055711_0001_1-tez-dag.pb.txt
2016-05-18 00:57:35,860 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533055711_0001_1
2016-05-18 00:57:35,861 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,861 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720] every/(1/1) ...mblyplatform/0_'field'_-1
2016-05-18 00:57:35 Running Dag: dag_1463533055711_0001_1
2016-05-18 00:57:35,870 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533055711_0001_1, submitTime=1463533055853
2016-05-18 00:57:35,871 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001/tez-conf.pb" } size: 36913 timestamp: 1463533055000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:35,871 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,871 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:35,872 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001/tez-conf.pb" } size: 36913 timestamp: 1463533055000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:35,872 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]
2016-05-18 00:57:35,872 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:35,872 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:35,872 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533055711_0001_1, initTime=1463533055872
2016-05-18 00:57:35,873 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533055711_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:35,873 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:35,874 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:DAG_STARTED]: dagID=dag_1463533055711_0001_1, startTime=1463533055873
2016-05-18 00:57:35,874 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533055711_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:35,874 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C1A4B5C41A274E4BBE1D860DD20400A6 : {35B7675B7AF14352B11EE86F77C4AC06={InputName=35B7675B7AF14352B11EE86F77C4AC06}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:35,874 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 35B7675B7AF14352B11EE86F77C4AC06, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:35,874 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,874 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,874 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,874 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,875 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:35,875 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]
2016-05-18 00:57:35,875 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...mblyplatform/0_'field'_-1, applicationId=application_1463533055711_0001, dagName=[9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720] every/(1/1) ...mblyplatform/0_'field'_-1
2016-05-18 00:57:35,877 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 35B7675B7AF14352B11EE86F77C4AC06 on vertex vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,878 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:35,879 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]
2016-05-18 00:57:35,879 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]
2016-05-18 00:57:35,879 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 35B7675B7AF14352B11EE86F77C4AC06 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:35,879 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533055711_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:35,879 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B], numAdditionalOutputs=1
2016-05-18 00:57:35,879 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=DADC897C02F249FF92FB78D0FD537ADD, vertex=vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:35,880 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:35,880 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:35,881 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:35,881 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:35,881 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 35B7675B7AF14352B11EE86F77C4AC06 on vertex vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,881 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D9E050C368B64CD682B65109CF95709B:DADC897C02F249FF92FB78D0FD537ADD using oldmapred API
2016-05-18 00:57:35,882 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=DADC897C02F249FF92FB78D0FD537ADD, vertexName=D9E050C368B64CD682B65109CF95709B, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:35,883 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D9E050C368B64CD682B65109CF95709B, vertexId=vertex_1463533055711_0001_1_01, initRequestedTime=1463533055875, initedTime=1463533055879, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:35,883 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:35,885 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:35,885 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {35B7675B7AF14352B11EE86F77C4AC06=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:35,886 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] parallelism set to 1
2016-05-18 00:57:35,886 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:35,887 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C1A4B5C41A274E4BBE1D860DD20400A6, vertexId=vertex_1463533055711_0001_1_00, initRequestedTime=1463533055874, initedTime=1463533055886, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:35,887 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] with distanceFromRoot: 0
2016-05-18 00:57:35,887 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:35,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533055711_0001_1_00, startRequestedTime=1463533055875, startedTime=1463533055888
2016-05-18 00:57:35,889 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533055711_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:35,889 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:35,889 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533055711_0001_1_00 for vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B] numStartedSources: 1 numSources: 1
2016-05-18 00:57:35,890 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B] with distanceFromRoot: 1
2016-05-18 00:57:35,889 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C1A4B5C41A274E4BBE1D860DD20400A6
2016-05-18 00:57:35,890 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533055711_0001_1_01, startRequestedTime=1463533055889, startedTime=1463533055890
2016-05-18 00:57:35,891 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:35,892 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,892 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D9E050C368B64CD682B65109CF95709B with 0 source tasks and 1 pending tasks
2016-05-18 00:57:35,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:TASK_STARTED]: vertexName=C1A4B5C41A274E4BBE1D860DD20400A6, taskId=task_1463533055711_0001_1_00_000000, scheduledTime=1463533055892, launchTime=1463533055892
2016-05-18 00:57:35,892 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,893 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C1A4B5C41A274E4BBE1D860DD20400A6 in vertex: D9E050C368B64CD682B65109CF95709B numBipartiteSourceTasks: 1
2016-05-18 00:57:35,895 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:35,896 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533055711_0001, with dag id: dag_1463533055711_0001_1
2016-05-18 00:57:35,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533055711_0001_00_000001, launchTime=1463533055896
2016-05-18 00:57:35,898 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533055711_0001_00_000001
2016-05-18 00:57:35,900 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533055711_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533055711_0001_1_00_000000_0
2016-05-18 00:57:35,900 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533055711_0001_1_00_000000_0] started. Is using containerId: [container_1463533055711_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:35,900 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C1A4B5C41A274E4BBE1D860DD20400A6, taskAttemptId=attempt_1463533055711_0001_1_00_000000_0, startTime=1463533055900, containerId=container_1463533055711_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:35 Starting to run new task attempt: attempt_1463533055711_0001_1_00_000000_0
2016-05-18 00:57:35,901 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533055711_0001_1_00_000000_0
2016-05-18 00:57:35,901 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,902 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:35,902 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720] every/(1/1) ...mblyplatform/0_'field'_-1, VertexName: C1A4B5C41A274E4BBE1D860DD20400A6, VertexParallelism: 1, TaskAttemptID:attempt_1463533055711_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=35B7675B7AF14352B11EE86F77C4AC06, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D9E050C368B64CD682B65109CF95709B, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:35,903 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:35,903 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533055711_0001_1_00_000000_0
2016-05-18 00:57:35,904 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533055711_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:35,904 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533055711_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:35,906 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:35,909 INFO input.MRInput (MRInput.java:initialize(440)) - 35B7675B7AF14352B11EE86F77C4AC06 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:35,909 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 35B7675B7AF14352B11EE86F77C4AC06
2016-05-18 00:57:35,911 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C1A4B5C41A274E4BBE1D860DD20400A6, ordinal: 0
2016-05-18 00:57:35,911 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C1A4B5C41A274E4BBE1D860DD20400A6, mem on start (mb), free: 247, total: 362, max: 672
2016-05-18 00:57:35,911 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:35,911 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:35,911 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:35,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:35,912 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D9E050C368B64CD682B65109CF95709B:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [35B7675B7AF14352B11EE86F77C4AC06:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 00:57:35,912 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:35,912 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:35,913 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:35,913 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D9E050C368B64CD682B65109CF95709B:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [35B7675B7AF14352B11EE86F77C4AC06:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 00:57:35,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:35,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 35B7675B7AF14352B11EE86F77C4AC06 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:35,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 35B7675B7AF14352B11EE86F77C4AC06
2016-05-18 00:57:35,914 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:35,914 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:35,914 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:35,915 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533055711_0001_1_00_000000_0
2016-05-18 00:57:35,917 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:35,917 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:35,917 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 35B7675B7AF14352B11EE86F77C4AC06 initialized RecordReader from event
2016-05-18 00:57:35,918 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 35B7675B7AF14352B11EE86F77C4AC06
2016-05-18 00:57:35,918 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 1D0D28CCE82D4480A3B4E41A0F3FBAC1
2016-05-18 00:57:35,918 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 1D0D28CCE82D4480A3B4E41A0F3FBAC1
2016-05-18 00:57:35,919 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:35,922 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:35,923 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D9E050C368B64CD682B65109CF95709B using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@34c74783, comparator=cascading.tuple.hadoop.util.TupleComparator@7280487, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:35,923 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D9E050C368B64CD682B65109CF95709B: , UsingHashComparator=false
2016-05-18 00:57:35,935 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:35,935 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:35,936 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D9E050C368B64CD682B65109CF95709B: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:35,936 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D9E050C368B64CD682B65109CF95709B: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:35,936 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:35,936 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C1A4B5C41A274E4BBE1D860DD20400A6, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:35,939 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533055711_0001_1_00_000000_0
2016-05-18 00:57:35,939 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C1A4B5C41A274E4BBE1D860DD20400A6, mem on close (mb), free: 145, total: 362, max: 672
2016-05-18 00:57:35,939 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D9E050C368B64CD682B65109CF95709B: Starting flush of map output
2016-05-18 00:57:35,939 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D9E050C368B64CD682B65109CF95709B: Span0.length = 20, perItem = 6
2016-05-18 00:57:35,940 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D9E050C368B64CD682B65109CF95709B: done sorting span=0, length=20, time=0
2016-05-18 00:57:35,940 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D9E050C368B64CD682B65109CF95709B: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:35,941 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D9E050C368B64CD682B65109CF95709B: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001_wd/localmode-local-dir/output/attempt_1463533055711_0001_1_00_000000_0_10117_0/file.out
2016-05-18 00:57:35,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:35 Completed running task attempt: attempt_1463533055711_0001_1_00_000000_0
2016-05-18 00:57:35,945 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533055711_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:35,945 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533055711_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:35,945 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533055711_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533055936, Process_Duration=3, Process_End_Time=1463533055939, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:35,947 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533055711_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:35,947 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:35,947 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:35,947 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,947 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C1A4B5C41A274E4BBE1D860DD20400A6, taskAttemptId=attempt_1463533055711_0001_1_00_000000_0, creationTime=1463533055892, allocationTime=1463533055896, startTime=1463533055900, finishTime=1463533055947, timeTaken=47, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:35,948 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533055711_0001_1_00_000000_0
2016-05-18 00:57:35,948 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:TASK_FINISHED]: vertexName=C1A4B5C41A274E4BBE1D860DD20400A6, taskId=task_1463533055711_0001_1_00_000000, startTime=1463533055900, finishTime=1463533055948, timeTaken=48, status=SUCCEEDED, successfulAttemptID=attempt_1463533055711_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533055936, Process_Duration=3, Process_End_Time=1463533055939, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:35,949 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533055711_0001_00_000001
2016-05-18 00:57:35,950 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:35,950 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:35,951 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:VERTEX_FINISHED]: vertexName=C1A4B5C41A274E4BBE1D860DD20400A6, vertexId=vertex_1463533055711_0001_1_00, initRequestedTime=1463533055874, initedTime=1463533055886, startRequestedTime=1463533055875, startedTime=1463533055888, finishTime=1463533055951, timeTaken=63, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533055936, Process_Duration=3, Process_End_Time=1463533055939, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533055900, firstTasksToStart=[ task_1463533055711_0001_1_00_000000 ], lastTaskFinishTime=1463533055947, lastTasksToFinish=[ task_1463533055711_0001_1_00_000000 ], minTaskDuration=47, maxTaskDuration=47, avgTaskDuration=47.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533055711_0001_1_00_000000 ], longestDurationTasks=[ task_1463533055711_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:35,951 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:35,952 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533055711_0001_00_000001, stoppedTime=1463533055952, exitStatus=0
2016-05-18 00:57:35,952 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B] attempt: attempt_1463533055711_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:35,952 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D9E050C368B64CD682B65109CF95709B
2016-05-18 00:57:35,952 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:35,953 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C1A4B5C41A274E4BBE1D860DD20400A6, destinationVertexName=D9E050C368B64CD682B65109CF95709B. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:35,953 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]
2016-05-18 00:57:35,953 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:35,954 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533055711_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:35,954 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:TASK_STARTED]: vertexName=D9E050C368B64CD682B65109CF95709B, taskId=task_1463533055711_0001_1_01_000000, scheduledTime=1463533055954, launchTime=1463533055954
2016-05-18 00:57:35,954 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]
2016-05-18 00:57:35,956 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533055711_0001_00_000002, launchTime=1463533055955
2016-05-18 00:57:35,956 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533055711_0001_00_000002
2016-05-18 00:57:35,957 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533055711_0001_1_01_000000_0] started. Is using containerId: [container_1463533055711_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:35,957 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D9E050C368B64CD682B65109CF95709B, taskAttemptId=attempt_1463533055711_0001_1_01_000000_0, startTime=1463533055956, containerId=container_1463533055711_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:35,957 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533055711_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533055711_0001_1_01_000000_0
2016-05-18 00:57:35 Starting to run new task attempt: attempt_1463533055711_0001_1_01_000000_0
2016-05-18 00:57:35,958 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533055711_0001_1_01_000000_0
2016-05-18 00:57:35,958 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,958 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:35,958 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720] every/(1/1) ...mblyplatform/0_'field'_-1, VertexName: D9E050C368B64CD682B65109CF95709B, VertexParallelism: 1, TaskAttemptID:attempt_1463533055711_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C1A4B5C41A274E4BBE1D860DD20400A6, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DADC897C02F249FF92FB78D0FD537ADD, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:35,958 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:35,959 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533055711_0001_1_01_000000_0
2016-05-18 00:57:35,959 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533055711_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]
2016-05-18 00:57:35,960 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533055711_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:35,961 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533055711_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:35,961 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533055711_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:35,961 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533055711_0001_00_000001, Exiting
2016-05-18 00:57:35,961 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533055711_0001_00_000001
2016-05-18 00:57:35,962 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533055711_0001_00_000001
2016-05-18 00:57:35,963 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:35,965 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533055711_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:35,965 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533055711_0001_00_000001 completed successfully
2016-05-18 00:57:35,966 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:35,969 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D9E050C368B64CD682B65109CF95709B, ordinal: 1
2016-05-18 00:57:35,969 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D9E050C368B64CD682B65109CF95709B, mem on start (mb), free: 140, total: 362, max: 672
2016-05-18 00:57:35,969 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:35,969 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:35,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:35,971 INFO output.MROutput (MROutput.java:initialize(421)) - DADC897C02F249FF92FB78D0FD537ADD: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:35,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:35,972 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DADC897C02F249FF92FB78D0FD537ADD:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [C1A4B5C41A274E4BBE1D860DD20400A6:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:35,972 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:35,972 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:35,972 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:35,972 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DADC897C02F249FF92FB78D0FD537ADD:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [C1A4B5C41A274E4BBE1D860DD20400A6:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:35,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:35,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C1A4B5C41A274E4BBE1D860DD20400A6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:35,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:35,973 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:35,973 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C1A4B5C41A274E4BBE1D860DD20400A6: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:35,974 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C1A4B5C41A274E4BBE1D860DD20400A6: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:35,974 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C1A4B5C41A274E4BBE1D860DD20400A6 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:35,976 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C1A4B5C41A274E4BBE1D860DD20400A6
2016-05-18 00:57:35,976 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:35,976 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533055711_0001_1_01_000000_0
2016-05-18 00:57:35,977 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C1A4B5C41A274E4BBE1D860DD20400A6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:35,980 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533055711_0001_1_00_000000_0_10117} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533055980, TimeTaken=2, Rate=0.09 MB/s
2016-05-18 00:57:35,980 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C1A4B5C41A274E4BBE1D860DD20400A6
2016-05-18 00:57:35,981 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:35,981 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:35,981 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 1D0D28CCE82D4480A3B4E41A0F3FBAC1
2016-05-18 00:57:35,982 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_-1"], id: DADC897C02F249FF92FB78D0FD537ADD
2016-05-18 00:57:35,982 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_-1"]
2016-05-18 00:57:35,982 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 1D0D28CCE82D4480A3B4E41A0F3FBAC1, for 1 inputs
2016-05-18 00:57:35,982 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C1A4B5C41A274E4BBE1D860DD20400A6, wasInterrupted=false
2016-05-18 00:57:35,984 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:35,985 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:35,986 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:35,986 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001_wd/localmode-local-dir/output/attempt_1463533055711_0001_1_00_000000_0_10117/file.out, len=181, isLocal=true
2016-05-18 00:57:35,986 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:35,987 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:35,987 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:35,987 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:35,987 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D9E050C368B64CD682B65109CF95709B, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:35,987 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C1A4B5C41A274E4BBE1D860DD20400A6
2016-05-18 00:57:35,988 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:35,988 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C1A4B5C41A274E4BBE1D860DD20400A6: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:35,989 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C1A4B5C41A274E4BBE1D860DD20400A6: Shuffle Runner thread complete
2016-05-18 00:57:35,993 INFO output.MROutput (MROutput.java:close(526)) - DADC897C02F249FF92FB78D0FD537ADD closed
2016-05-18 00:57:35,994 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533055711_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:35,994 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533055711_0001_1_01_000000_0
2016-05-18 00:57:35,994 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D9E050C368B64CD682B65109CF95709B, mem on close (mb), free: 136, total: 362, max: 672
2016-05-18 00:57:35,995 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C1A4B5C41A274E4BBE1D860DD20400A6
2016-05-18 00:57:35,995 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C1A4B5C41A274E4BBE1D860DD20400A6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:35,995 INFO output.MROutput (MROutput.java:close(526)) - DADC897C02F249FF92FB78D0FD537ADD closed
2016-05-18 00:57:35,995 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:35 Completed running task attempt: attempt_1463533055711_0001_1_01_000000_0
2016-05-18 00:57:35,995 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533055711_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:35,996 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533055711_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:35,996 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533055711_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533055987, Process_Duration=7, Process_End_Time=1463533055994, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:35,997 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533055711_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:35,998 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:35,998 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:35,998 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:35,998 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D9E050C368B64CD682B65109CF95709B, taskAttemptId=attempt_1463533055711_0001_1_01_000000_0, creationTime=1463533055954, allocationTime=1463533055955, startTime=1463533055956, finishTime=1463533055998, timeTaken=42, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:35,998 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533055711_0001_00_000002
2016-05-18 00:57:35,998 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533055711_0001_1_01_000000_0
2016-05-18 00:57:35,999 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:TASK_FINISHED]: vertexName=D9E050C368B64CD682B65109CF95709B, taskId=task_1463533055711_0001_1_01_000000, startTime=1463533055956, finishTime=1463533055998, timeTaken=42, status=SUCCEEDED, successfulAttemptID=attempt_1463533055711_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533055987, Process_Duration=7, Process_End_Time=1463533055994, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:36,000 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:36,000 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B]
2016-05-18 00:57:36,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:VERTEX_FINISHED]: vertexName=D9E050C368B64CD682B65109CF95709B, vertexId=vertex_1463533055711_0001_1_01, initRequestedTime=1463533055875, initedTime=1463533055879, startRequestedTime=1463533055889, startedTime=1463533055890, finishTime=1463533056000, timeTaken=110, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533055987, Process_Duration=7, Process_End_Time=1463533055994, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533055956, firstTasksToStart=[ task_1463533055711_0001_1_01_000000 ], lastTaskFinishTime=1463533055998, lastTasksToFinish=[ task_1463533055711_0001_1_01_000000 ], minTaskDuration=42, maxTaskDuration=42, avgTaskDuration=42.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533055711_0001_1_01_000000 ], longestDurationTasks=[ task_1463533055711_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:36,001 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:36,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533055711_0001_00_000002, stoppedTime=1463533056001, exitStatus=0
2016-05-18 00:57:36,002 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:36,002 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:36,002 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533055711_0001_1_00 [C1A4B5C41A274E4BBE1D860DD20400A6]
2016-05-18 00:57:36,002 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533055711_0001_1
2016-05-18 00:57:36,002 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533055711_0001_1
2016-05-18 00:57:36,002 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533055711_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:36,002 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: DADC897C02F249FF92FB78D0FD537ADD for vertex: vertex_1463533055711_0001_1_01 [D9E050C368B64CD682B65109CF95709B], outputName: DADC897C02F249FF92FB78D0FD537ADD
2016-05-18 00:57:36,002 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533055711_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:36,009 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:DADC897C02F249FF92FB78D0FD537ADD of vertex/vertexGroup:D9E050C368B64CD682B65109CF95709B isVertexGroupOutput:false
2016-05-18 00:57:36,009 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:36,009 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533055711_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:36,009 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533055711_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:36,009 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533055711_0001_00_000002, Exiting
2016-05-18 00:57:36,009 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533055711_0001_00_000002
2016-05-18 00:57:36,009 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533055711_0001_00_000002
2016-05-18 00:57:36,010 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533055711_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:36,010 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533055711_0001_00_000002 completed successfully
2016-05-18 00:57:36,011 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533055711_0001_1, queueSize=0
2016-05-18 00:57:36,011 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533055711_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533055711_0001_1, startTime=1463533055873, finishTime=1463533056009, timeTaken=136, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=230, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066111923, Process_Duration=10, Process_End_Time=2927066111933, Read_Duration=2, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:36,011 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533055711_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:36,011 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533055711_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:36 Completed Dag: dag_1463533055711_0001_1
2016-05-18 00:57:36,012 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533055711_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:36,012 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533055711_0001_1_post
2016-05-18 00:57:36,012 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:36,012 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:36,012 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:36,012 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720] every/(1/1) ...mblyplatform/0_'field'_-1, with id=dag_1463533055711_0001_1
2016-05-18 00:57:36,012 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720] every/(1/1) ...mblyplatform/0_'field'_-1, with id=dag_1463533055711_0001_1
2016-05-18 00:57:36,022 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...mblyplatform/0_'field'_-1, applicationId=application_1463533055711_0001
2016-05-18 00:57:36,024 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:36,024 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:36,024 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:36,024 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:36,025 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:36,025 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:36,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:36,025 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:36,025 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:36,026 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:36,026 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:36,026 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:36,027 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:36,027 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:36,032 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9D3C71C514354487AE1E7725FD78E639/B930527B959B4C28B6877B92F2928720/.tez/application_1463533055711_0001
2016-05-18 00:57:36,032 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_-1/_temporary
2016-05-18 00:57:36,033 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.340
2016-05-18 00:57:36,127 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_'field'_-1
2016-05-18 00:57:36,165 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:36,166 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:36,203 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.008
2016-05-18 00:57:36,211 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.016
2016-05-18 00:57:36,211 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:36,211 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:36,212 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:36,228 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:36,235 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_'field'
2016-05-18 00:57:36,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:36,287 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:36,287 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_'field'"]
2016-05-18 00:57:36,287 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:36,287 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:36,287 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:36,288 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...latform/0_'field'_'field'
2016-05-18 00:57:36,288 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:36,300 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:36,304 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:36,305 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:36,305 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:36,313 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001 doesn't exist and is created
2016-05-18 00:57:36,315 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:36,315 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:36,329 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:36,329 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:36,331 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001_wd
2016-05-18 00:57:36,341 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533056305_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:36,341 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:36,342 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:36,343 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:36,343 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:36,343 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:36,343 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:36,344 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:36,345 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:36,346 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:36,346 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:36,346 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:36,346 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:36,346 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:36,346 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:36,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:36,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:36,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:36,347 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:36,347 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:36,349 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:36,350 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:36,350 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:36,350 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001_wd/localmode-log-dir
2016-05-18 00:57:36,351 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001_wd/localmode-log-dir/history.txt.appattempt_1463533056305_0001_000000, maxErrors=10
2016-05-18 00:57:36,351 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:36,351 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533056305_0001_000000, appSubmitTime=1463533056341, launchTime=1463533056341
2016-05-18 00:57:36,355 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:36,357 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:60723
2016-05-18 00:57:36,361 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533056305
2016-05-18 00:57:36,363 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533056305_0001_000000, startTime=1463533056363
2016-05-18 00:57:36,363 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:36,429 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:36,430 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:36,430 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...latform/0_'field'_'field', applicationId=application_1463533056305_0001, dagName=[BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10] every/(1/1) ...latform/0_'field'_'field'
2016-05-18 00:57:36,446 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10] every/(1/1) ...latform/0_'field'_'field'
2016-05-18 00:57:36,446 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:36,447 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533056305_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001_wd/localmode-log-dir/dag_1463533056305_0001_1.dot
2016-05-18 00:57:36,447 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001_wd/localmode-log-dir/dag_1463533056305_0001_1-tez-dag.pb.txt
2016-05-18 00:57:36,454 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533056305_0001_1
2016-05-18 00:57:36,454 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:36,454 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10] every/(1/1) ...latform/0_'field'_'field'
2016-05-18 00:57:36 Running Dag: dag_1463533056305_0001_1
2016-05-18 00:57:36,464 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533056305_0001_1, submitTime=1463533056446
2016-05-18 00:57:36,466 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001/tez-conf.pb" } size: 36913 timestamp: 1463533056000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:36,466 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,467 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:36,467 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001/tez-conf.pb" } size: 36913 timestamp: 1463533056000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:36,467 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]
2016-05-18 00:57:36,468 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:36,468 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:36,469 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533056305_0001_1, initTime=1463533056469
2016-05-18 00:57:36,469 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533056305_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:36,469 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:36,469 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:DAG_STARTED]: dagID=dag_1463533056305_0001_1, startTime=1463533056469
2016-05-18 00:57:36,469 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533056305_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:36,470 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D3700F16957C4DBF84789458E8264A90 : {AE9D2ECCF0424780BED7F74A21502874={InputName=AE9D2ECCF0424780BED7F74A21502874}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:36,470 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: AE9D2ECCF0424780BED7F74A21502874, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:36,470 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,470 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,470 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,471 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,471 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:36,471 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...latform/0_'field'_'field', applicationId=application_1463533056305_0001, dagName=[BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10] every/(1/1) ...latform/0_'field'_'field'
2016-05-18 00:57:36,471 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]
2016-05-18 00:57:36,471 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: AE9D2ECCF0424780BED7F74A21502874 on vertex vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,473 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input AE9D2ECCF0424780BED7F74A21502874 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:36,474 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:36,474 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]
2016-05-18 00:57:36,475 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]
2016-05-18 00:57:36,475 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:36,475 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:36,475 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:36,476 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533056305_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:36,476 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5], numAdditionalOutputs=1
2016-05-18 00:57:36,476 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=38AD306A620B4864BFF0F7C567FE53E1, vertex=vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:36,476 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:36,476 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: AE9D2ECCF0424780BED7F74A21502874 on vertex vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,477 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F72C6A44F0974DFD827231D2149168B5:38AD306A620B4864BFF0F7C567FE53E1 using oldmapred API
2016-05-18 00:57:36,478 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=38AD306A620B4864BFF0F7C567FE53E1, vertexName=F72C6A44F0974DFD827231D2149168B5, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:36,478 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F72C6A44F0974DFD827231D2149168B5, vertexId=vertex_1463533056305_0001_1_01, initRequestedTime=1463533056471, initedTime=1463533056476, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:36,478 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:36,479 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:36,479 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {AE9D2ECCF0424780BED7F74A21502874=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:36,480 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] parallelism set to 1
2016-05-18 00:57:36,480 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:36,480 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D3700F16957C4DBF84789458E8264A90, vertexId=vertex_1463533056305_0001_1_00, initRequestedTime=1463533056470, initedTime=1463533056480, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:36,480 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] with distanceFromRoot: 0
2016-05-18 00:57:36,481 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:36,481 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533056305_0001_1_00, startRequestedTime=1463533056471, startedTime=1463533056481
2016-05-18 00:57:36,481 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533056305_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:36,482 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:36,482 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533056305_0001_1_00 for vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5] numStartedSources: 1 numSources: 1
2016-05-18 00:57:36,482 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5] with distanceFromRoot: 1
2016-05-18 00:57:36,482 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D3700F16957C4DBF84789458E8264A90
2016-05-18 00:57:36,483 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,483 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533056305_0001_1_01, startRequestedTime=1463533056482, startedTime=1463533056482
2016-05-18 00:57:36,483 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:36,486 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F72C6A44F0974DFD827231D2149168B5 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:36,486 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:TASK_STARTED]: vertexName=D3700F16957C4DBF84789458E8264A90, taskId=task_1463533056305_0001_1_00_000000, scheduledTime=1463533056486, launchTime=1463533056486
2016-05-18 00:57:36,489 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D3700F16957C4DBF84789458E8264A90 in vertex: F72C6A44F0974DFD827231D2149168B5 numBipartiteSourceTasks: 1
2016-05-18 00:57:36,489 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,490 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:36,492 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533056305_0001_00_000001, launchTime=1463533056492
2016-05-18 00:57:36,493 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533056305_0001_00_000001
2016-05-18 00:57:36,493 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533056305_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533056305_0001_1_00_000000_0
2016-05-18 00:57:36,493 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533056305_0001_1_00_000000_0] started. Is using containerId: [container_1463533056305_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:36 Starting to run new task attempt: attempt_1463533056305_0001_1_00_000000_0
2016-05-18 00:57:36,494 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D3700F16957C4DBF84789458E8264A90, taskAttemptId=attempt_1463533056305_0001_1_00_000000_0, startTime=1463533056493, containerId=container_1463533056305_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:36,494 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533056305_0001_1_00_000000_0
2016-05-18 00:57:36,494 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:36,494 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:36,494 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10] every/(1/1) ...latform/0_'field'_'field', VertexName: D3700F16957C4DBF84789458E8264A90, VertexParallelism: 1, TaskAttemptID:attempt_1463533056305_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AE9D2ECCF0424780BED7F74A21502874, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F72C6A44F0974DFD827231D2149168B5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:36,494 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:36,495 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533056305_0001_1_00_000000_0
2016-05-18 00:57:36,496 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533056305_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:36,496 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533056305_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:36,497 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533056305_0001, with dag id: dag_1463533056305_0001_1
2016-05-18 00:57:36,501 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:36,503 INFO input.MRInput (MRInput.java:initialize(440)) - AE9D2ECCF0424780BED7F74A21502874 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:36,503 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: AE9D2ECCF0424780BED7F74A21502874
2016-05-18 00:57:36,506 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D3700F16957C4DBF84789458E8264A90, ordinal: 0
2016-05-18 00:57:36,506 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D3700F16957C4DBF84789458E8264A90, mem on start (mb), free: 231, total: 364, max: 672
2016-05-18 00:57:36,506 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:36,506 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:36,506 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:36,507 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:36,510 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[AE9D2ECCF0424780BED7F74A21502874:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F72C6A44F0974DFD827231D2149168B5:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:36,510 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:36,510 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:36,510 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:36,510 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[AE9D2ECCF0424780BED7F74A21502874:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F72C6A44F0974DFD827231D2149168B5:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:36,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:36,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AE9D2ECCF0424780BED7F74A21502874 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:36,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:36,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:36,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AE9D2ECCF0424780BED7F74A21502874
2016-05-18 00:57:36,511 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:36,511 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533056305_0001_1_00_000000_0
2016-05-18 00:57:36,515 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:36,517 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: AE9D2ECCF0424780BED7F74A21502874
2016-05-18 00:57:36,517 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:36,518 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - AE9D2ECCF0424780BED7F74A21502874 initialized RecordReader from event
2016-05-18 00:57:36,517 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 248682482F3C4521BFD694904437FCB7
2016-05-18 00:57:36,518 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 248682482F3C4521BFD694904437FCB7
2016-05-18 00:57:36,520 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:36,522 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:36,522 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F72C6A44F0974DFD827231D2149168B5 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@3732eafc, comparator=cascading.tuple.hadoop.util.TupleComparator@6fe162b3, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:36,522 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F72C6A44F0974DFD827231D2149168B5: , UsingHashComparator=false
2016-05-18 00:57:36,536 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:36,536 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:36,536 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F72C6A44F0974DFD827231D2149168B5: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:36,537 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F72C6A44F0974DFD827231D2149168B5: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:36,537 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:36,538 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D3700F16957C4DBF84789458E8264A90, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:36,541 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533056305_0001_1_00_000000_0
2016-05-18 00:57:36,542 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D3700F16957C4DBF84789458E8264A90, mem on close (mb), free: 128, total: 364, max: 672
2016-05-18 00:57:36,543 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F72C6A44F0974DFD827231D2149168B5: Starting flush of map output
2016-05-18 00:57:36,543 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F72C6A44F0974DFD827231D2149168B5: Span0.length = 20, perItem = 6
2016-05-18 00:57:36,544 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F72C6A44F0974DFD827231D2149168B5: done sorting span=0, length=20, time=1
2016-05-18 00:57:36,545 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F72C6A44F0974DFD827231D2149168B5: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:36,546 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F72C6A44F0974DFD827231D2149168B5: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001_wd/localmode-local-dir/output/attempt_1463533056305_0001_1_00_000000_0_10123_0/file.out
2016-05-18 00:57:36,549 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:36 Completed running task attempt: attempt_1463533056305_0001_1_00_000000_0
2016-05-18 00:57:36,550 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533056305_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:36,550 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533056305_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:36,551 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533056305_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533056538, Process_Duration=3, Process_End_Time=1463533056541, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:36,552 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533056305_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:36,552 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:36,552 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:36,552 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:36,553 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D3700F16957C4DBF84789458E8264A90, taskAttemptId=attempt_1463533056305_0001_1_00_000000_0, creationTime=1463533056489, allocationTime=1463533056491, startTime=1463533056493, finishTime=1463533056553, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:36,553 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533056305_0001_1_00_000000_0
2016-05-18 00:57:36,554 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:TASK_FINISHED]: vertexName=D3700F16957C4DBF84789458E8264A90, taskId=task_1463533056305_0001_1_00_000000, startTime=1463533056493, finishTime=1463533056553, timeTaken=60, status=SUCCEEDED, successfulAttemptID=attempt_1463533056305_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533056538, Process_Duration=3, Process_End_Time=1463533056541, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:36,554 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:36,554 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533056305_0001_00_000001
2016-05-18 00:57:36,554 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,556 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:VERTEX_FINISHED]: vertexName=D3700F16957C4DBF84789458E8264A90, vertexId=vertex_1463533056305_0001_1_00, initRequestedTime=1463533056470, initedTime=1463533056480, startRequestedTime=1463533056471, startedTime=1463533056481, finishTime=1463533056555, timeTaken=74, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533056538, Process_Duration=3, Process_End_Time=1463533056541, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533056493, firstTasksToStart=[ task_1463533056305_0001_1_00_000000 ], lastTaskFinishTime=1463533056553, lastTasksToFinish=[ task_1463533056305_0001_1_00_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533056305_0001_1_00_000000 ], longestDurationTasks=[ task_1463533056305_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:36,556 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:36,556 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5] attempt: attempt_1463533056305_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:36,557 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F72C6A44F0974DFD827231D2149168B5
2016-05-18 00:57:36,557 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D3700F16957C4DBF84789458E8264A90, destinationVertexName=F72C6A44F0974DFD827231D2149168B5. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:36,557 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]
2016-05-18 00:57:36,557 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533056305_0001_00_000001, stoppedTime=1463533056557, exitStatus=0
2016-05-18 00:57:36,561 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:36,561 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:36,561 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:TASK_STARTED]: vertexName=F72C6A44F0974DFD827231D2149168B5, taskId=task_1463533056305_0001_1_01_000000, scheduledTime=1463533056561, launchTime=1463533056561
2016-05-18 00:57:36,562 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533056305_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:36,563 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]
2016-05-18 00:57:36,565 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533056305_0001_00_000002, launchTime=1463533056565
2016-05-18 00:57:36,565 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533056305_0001_00_000002
2016-05-18 00:57:36,566 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533056305_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:36,566 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533056305_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:36,566 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533056305_0001_00_000001, Exiting
2016-05-18 00:57:36,566 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533056305_0001_00_000001
2016-05-18 00:57:36,566 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533056305_0001_00_000001
2016-05-18 00:57:36,566 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533056305_0001_1_01_000000_0] started. Is using containerId: [container_1463533056305_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:36,566 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F72C6A44F0974DFD827231D2149168B5, taskAttemptId=attempt_1463533056305_0001_1_01_000000_0, startTime=1463533056566, containerId=container_1463533056305_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:36,567 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533056305_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533056305_0001_1_01_000000_0
2016-05-18 00:57:36,567 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533056305_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:36,568 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533056305_0001_00_000001 completed successfully
2016-05-18 00:57:36 Starting to run new task attempt: attempt_1463533056305_0001_1_01_000000_0
2016-05-18 00:57:36,568 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533056305_0001_1_01_000000_0
2016-05-18 00:57:36,568 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:36,569 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:36,569 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10] every/(1/1) ...latform/0_'field'_'field', VertexName: F72C6A44F0974DFD827231D2149168B5, VertexParallelism: 1, TaskAttemptID:attempt_1463533056305_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D3700F16957C4DBF84789458E8264A90, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=38AD306A620B4864BFF0F7C567FE53E1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:36,569 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:36,570 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533056305_0001_1_01_000000_0
2016-05-18 00:57:36,570 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533056305_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]
2016-05-18 00:57:36,570 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533056305_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:36,574 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:36,575 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:36,580 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F72C6A44F0974DFD827231D2149168B5, ordinal: 1
2016-05-18 00:57:36,580 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F72C6A44F0974DFD827231D2149168B5, mem on start (mb), free: 124, total: 364, max: 672
2016-05-18 00:57:36,580 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:36,580 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:36,580 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:36,581 INFO output.MROutput (MROutput.java:initialize(421)) - 38AD306A620B4864BFF0F7C567FE53E1: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:36,581 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:36,581 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[38AD306A620B4864BFF0F7C567FE53E1:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [D3700F16957C4DBF84789458E8264A90:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:36,582 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:36,582 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:36,582 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:36,582 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[38AD306A620B4864BFF0F7C567FE53E1:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [D3700F16957C4DBF84789458E8264A90:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:36,582 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:36,582 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D3700F16957C4DBF84789458E8264A90 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:36,583 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:36,583 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:36,583 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D3700F16957C4DBF84789458E8264A90: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:36,583 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D3700F16957C4DBF84789458E8264A90: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:36,584 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D3700F16957C4DBF84789458E8264A90 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:36,586 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D3700F16957C4DBF84789458E8264A90
2016-05-18 00:57:36,586 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:36,587 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533056305_0001_1_01_000000_0
2016-05-18 00:57:36,588 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D3700F16957C4DBF84789458E8264A90: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:36,589 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:36,590 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533056305_0001_1_00_000000_0_10123} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533056590, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:36,590 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D3700F16957C4DBF84789458E8264A90
2016-05-18 00:57:36,590 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 248682482F3C4521BFD694904437FCB7
2016-05-18 00:57:36,590 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:36,590 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_'field'"], id: 38AD306A620B4864BFF0F7C567FE53E1
2016-05-18 00:57:36,591 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_'field'"]
2016-05-18 00:57:36,591 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D3700F16957C4DBF84789458E8264A90, wasInterrupted=false
2016-05-18 00:57:36,592 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 248682482F3C4521BFD694904437FCB7, for 1 inputs
2016-05-18 00:57:36,592 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:36,594 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:36,595 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:36,595 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001_wd/localmode-local-dir/output/attempt_1463533056305_0001_1_00_000000_0_10123/file.out, len=181, isLocal=true
2016-05-18 00:57:36,595 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:36,595 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:36,595 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:36,596 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:36,596 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D3700F16957C4DBF84789458E8264A90
2016-05-18 00:57:36,596 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D3700F16957C4DBF84789458E8264A90: Shuffle Runner thread complete
2016-05-18 00:57:36,596 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F72C6A44F0974DFD827231D2149168B5, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:36,597 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:36,597 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D3700F16957C4DBF84789458E8264A90: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:36,602 INFO output.MROutput (MROutput.java:close(526)) - 38AD306A620B4864BFF0F7C567FE53E1 closed
2016-05-18 00:57:36,603 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533056305_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:36,603 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533056305_0001_1_01_000000_0
2016-05-18 00:57:36,603 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F72C6A44F0974DFD827231D2149168B5, mem on close (mb), free: 119, total: 364, max: 672
2016-05-18 00:57:36,604 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D3700F16957C4DBF84789458E8264A90
2016-05-18 00:57:36,604 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D3700F16957C4DBF84789458E8264A90: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:36,604 INFO output.MROutput (MROutput.java:close(526)) - 38AD306A620B4864BFF0F7C567FE53E1 closed
2016-05-18 00:57:36,604 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:36 Completed running task attempt: attempt_1463533056305_0001_1_01_000000_0
2016-05-18 00:57:36,605 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533056305_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:36,605 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533056305_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:36,605 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533056305_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533056596, Process_Duration=7, Process_End_Time=1463533056603, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:36,606 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533056305_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:36,606 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:36,606 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:36,606 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:36,606 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533056305_0001_00_000002
2016-05-18 00:57:36,606 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F72C6A44F0974DFD827231D2149168B5, taskAttemptId=attempt_1463533056305_0001_1_01_000000_0, creationTime=1463533056562, allocationTime=1463533056564, startTime=1463533056566, finishTime=1463533056606, timeTaken=40, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:36,607 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533056305_0001_1_01_000000_0
2016-05-18 00:57:36,607 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:TASK_FINISHED]: vertexName=F72C6A44F0974DFD827231D2149168B5, taskId=task_1463533056305_0001_1_01_000000, startTime=1463533056566, finishTime=1463533056607, timeTaken=41, status=SUCCEEDED, successfulAttemptID=attempt_1463533056305_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533056596, Process_Duration=7, Process_End_Time=1463533056603, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:36,607 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:36,607 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5]
2016-05-18 00:57:36,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:VERTEX_FINISHED]: vertexName=F72C6A44F0974DFD827231D2149168B5, vertexId=vertex_1463533056305_0001_1_01, initRequestedTime=1463533056471, initedTime=1463533056476, startRequestedTime=1463533056482, startedTime=1463533056482, finishTime=1463533056607, timeTaken=125, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533056596, Process_Duration=7, Process_End_Time=1463533056603, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533056566, firstTasksToStart=[ task_1463533056305_0001_1_01_000000 ], lastTaskFinishTime=1463533056606, lastTasksToFinish=[ task_1463533056305_0001_1_01_000000 ], minTaskDuration=40, maxTaskDuration=40, avgTaskDuration=40.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533056305_0001_1_01_000000 ], longestDurationTasks=[ task_1463533056305_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:36,608 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:36,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533056305_0001_00_000002, stoppedTime=1463533056608, exitStatus=0
2016-05-18 00:57:36,608 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:36,608 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:36,608 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533056305_0001_1_00 [D3700F16957C4DBF84789458E8264A90]
2016-05-18 00:57:36,609 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533056305_0001_1
2016-05-18 00:57:36,609 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533056305_0001_1
2016-05-18 00:57:36,609 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533056305_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:36,609 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 38AD306A620B4864BFF0F7C567FE53E1 for vertex: vertex_1463533056305_0001_1_01 [F72C6A44F0974DFD827231D2149168B5], outputName: 38AD306A620B4864BFF0F7C567FE53E1
2016-05-18 00:57:36,609 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533056305_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:36,616 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:38AD306A620B4864BFF0F7C567FE53E1 of vertex/vertexGroup:F72C6A44F0974DFD827231D2149168B5 isVertexGroupOutput:false
2016-05-18 00:57:36,616 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:36,617 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533056305_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:36,617 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533056305_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:36,617 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533056305_0001_00_000002, Exiting
2016-05-18 00:57:36,617 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533056305_0001_00_000002
2016-05-18 00:57:36,617 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533056305_0001_00_000002
2016-05-18 00:57:36,617 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533056305_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:36,618 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533056305_0001_00_000002 completed successfully
2016-05-18 00:57:36,618 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533056305_0001_1, queueSize=0
2016-05-18 00:57:36,618 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056305_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533056305_0001_1, startTime=1463533056469, finishTime=1463533056616, timeTaken=147, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=250, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066113134, Process_Duration=10, Process_End_Time=2927066113144, Read_Duration=4, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:36,618 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533056305_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:36,618 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533056305_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:36 Completed Dag: dag_1463533056305_0001_1
2016-05-18 00:57:36,619 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533056305_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:36,619 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533056305_0001_1_post
2016-05-18 00:57:36,619 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:36,619 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:36,619 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:36,619 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10] every/(1/1) ...latform/0_'field'_'field', with id=dag_1463533056305_0001_1
2016-05-18 00:57:36,619 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10] every/(1/1) ...latform/0_'field'_'field', with id=dag_1463533056305_0001_1
2016-05-18 00:57:36,630 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...latform/0_'field'_'field', applicationId=application_1463533056305_0001
2016-05-18 00:57:36,631 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:36,632 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:36,632 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:36,632 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:36,632 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:36,632 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:36,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:36,634 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:36,634 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:36,634 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:36,634 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:36,634 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:36,634 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:36,635 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:36,640 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/BF15437701AD4978B38EBEDE15E1C657/CCBD7B7433164AED83BE83030AD85A10/.tez/application_1463533056305_0001
2016-05-18 00:57:36,641 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_'field'/_temporary
2016-05-18 00:57:36,641 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.355
2016-05-18 00:57:36,720 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_'field'_'field'
2016-05-18 00:57:36,742 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:36,743 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:36,778 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.010
2016-05-18 00:57:36,791 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.023
2016-05-18 00:57:36,791 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:36,791 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:36,792 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:36,808 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:36,817 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_RESULTS
2016-05-18 00:57:36,883 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:36,883 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:36,884 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_RESULTS"]
2016-05-18 00:57:36,884 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:36,884 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:36,885 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:36,886 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...latform/0_'field'_RESULTS
2016-05-18 00:57:36,886 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:36,906 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:36,912 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:36,912 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:36,912 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:36,925 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001 doesn't exist and is created
2016-05-18 00:57:36,942 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:36,943 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:36,943 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001_wd
2016-05-18 00:57:36,953 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533056912_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:36,953 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:36,953 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:36,954 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:36,954 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:36,955 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:36,955 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:36,955 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:36,955 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:36,956 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:36,957 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:36,957 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:36,958 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:36,958 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:36,958 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:36,958 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:36,958 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:36,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:36,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:36,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:36,959 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:36,959 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:36,960 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:36,961 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:36,961 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:36,962 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001_wd/localmode-log-dir
2016-05-18 00:57:36,962 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001_wd/localmode-log-dir/history.txt.appattempt_1463533056912_0001_000000, maxErrors=10
2016-05-18 00:57:36,962 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:36,962 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533056912_0001_000000, appSubmitTime=1463533056953, launchTime=1463533056953
2016-05-18 00:57:36,966 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:36,969 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:45525
2016-05-18 00:57:36,969 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533056912
2016-05-18 00:57:36,971 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533056912_0001_000000, startTime=1463533056971
2016-05-18 00:57:36,971 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:37,043 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:37,043 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:37,043 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...latform/0_'field'_RESULTS, applicationId=application_1463533056912_0001, dagName=[B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E] every/(1/1) ...latform/0_'field'_RESULTS
2016-05-18 00:57:37,059 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E] every/(1/1) ...latform/0_'field'_RESULTS
2016-05-18 00:57:37,059 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:37,060 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533056912_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001_wd/localmode-log-dir/dag_1463533056912_0001_1.dot
2016-05-18 00:57:37,061 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001_wd/localmode-log-dir/dag_1463533056912_0001_1-tez-dag.pb.txt
2016-05-18 00:57:37,067 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533056912_0001_1
2016-05-18 00:57:37,067 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,067 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E] every/(1/1) ...latform/0_'field'_RESULTS
2016-05-18 00:57:37 Running Dag: dag_1463533056912_0001_1
2016-05-18 00:57:37,077 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533056912_0001_1, submitTime=1463533057059
2016-05-18 00:57:37,078 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001/tez-conf.pb" } size: 36913 timestamp: 1463533056000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:37,079 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,079 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:37,080 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001/tez-conf.pb" } size: 36913 timestamp: 1463533056000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:37,080 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]
2016-05-18 00:57:37,081 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:37,081 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:37,082 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533056912_0001_1, initTime=1463533057082
2016-05-18 00:57:37,082 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533056912_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:37,083 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:37,083 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:DAG_STARTED]: dagID=dag_1463533056912_0001_1, startTime=1463533057083
2016-05-18 00:57:37,083 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533056912_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:37,083 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D823A53DDF454BA79F82290F84B0DC16 : {BD9CFB51E38649ED9C2AD4EBDAD2AE42={InputName=BD9CFB51E38649ED9C2AD4EBDAD2AE42}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:37,083 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: BD9CFB51E38649ED9C2AD4EBDAD2AE42, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:37,083 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,083 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,083 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,084 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,084 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:37,084 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]
2016-05-18 00:57:37,085 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...latform/0_'field'_RESULTS, applicationId=application_1463533056912_0001, dagName=[B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E] every/(1/1) ...latform/0_'field'_RESULTS
2016-05-18 00:57:37,086 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: BD9CFB51E38649ED9C2AD4EBDAD2AE42 on vertex vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,087 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:37,087 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]
2016-05-18 00:57:37,088 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]
2016-05-18 00:57:37,088 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533056912_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:37,088 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218], numAdditionalOutputs=1
2016-05-18 00:57:37,089 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=A318480F527B4CC0986FF8C27EE9DCA2, vertex=vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:37,089 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input BD9CFB51E38649ED9C2AD4EBDAD2AE42 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:37,090 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for BF54167049CC4061B97304731A9EC218:A318480F527B4CC0986FF8C27EE9DCA2 using oldmapred API
2016-05-18 00:57:37,091 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=A318480F527B4CC0986FF8C27EE9DCA2, vertexName=BF54167049CC4061B97304731A9EC218, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:37,091 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:37,091 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:37,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BF54167049CC4061B97304731A9EC218, vertexId=vertex_1463533056912_0001_1_01, initRequestedTime=1463533057084, initedTime=1463533057088, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:37,092 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:37,092 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:37,093 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:37,093 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: BD9CFB51E38649ED9C2AD4EBDAD2AE42 on vertex vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,094 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:37,094 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {BD9CFB51E38649ED9C2AD4EBDAD2AE42=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:37,095 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] parallelism set to 1
2016-05-18 00:57:37,095 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:37,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D823A53DDF454BA79F82290F84B0DC16, vertexId=vertex_1463533056912_0001_1_00, initRequestedTime=1463533057083, initedTime=1463533057096, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:37,096 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] with distanceFromRoot: 0
2016-05-18 00:57:37,096 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:37,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533056912_0001_1_00, startRequestedTime=1463533057084, startedTime=1463533057096
2016-05-18 00:57:37,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533056912_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:37,097 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:37,097 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533056912_0001_1_00 for vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218] numStartedSources: 1 numSources: 1
2016-05-18 00:57:37,097 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218] with distanceFromRoot: 1
2016-05-18 00:57:37,098 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533056912_0001_1_01, startRequestedTime=1463533057097, startedTime=1463533057097
2016-05-18 00:57:37,098 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:37,100 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: BF54167049CC4061B97304731A9EC218 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:37,101 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D823A53DDF454BA79F82290F84B0DC16 in vertex: BF54167049CC4061B97304731A9EC218 numBipartiteSourceTasks: 1
2016-05-18 00:57:37,102 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D823A53DDF454BA79F82290F84B0DC16
2016-05-18 00:57:37,102 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,103 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:TASK_STARTED]: vertexName=D823A53DDF454BA79F82290F84B0DC16, taskId=task_1463533056912_0001_1_00_000000, scheduledTime=1463533057102, launchTime=1463533057102
2016-05-18 00:57:37,103 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,104 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:37,105 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533056912_0001_00_000001, launchTime=1463533057105
2016-05-18 00:57:37,106 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533056912_0001_00_000001
2016-05-18 00:57:37,106 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533056912_0001_1_00_000000_0] started. Is using containerId: [container_1463533056912_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:37,107 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D823A53DDF454BA79F82290F84B0DC16, taskAttemptId=attempt_1463533056912_0001_1_00_000000_0, startTime=1463533057106, containerId=container_1463533056912_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:37,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533056912_0001, with dag id: dag_1463533056912_0001_1
2016-05-18 00:57:37,107 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533056912_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533056912_0001_1_00_000000_0
2016-05-18 00:57:37 Starting to run new task attempt: attempt_1463533056912_0001_1_00_000000_0
2016-05-18 00:57:37,108 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533056912_0001_1_00_000000_0
2016-05-18 00:57:37,108 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,108 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:37,108 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E] every/(1/1) ...latform/0_'field'_RESULTS, VertexName: D823A53DDF454BA79F82290F84B0DC16, VertexParallelism: 1, TaskAttemptID:attempt_1463533056912_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BD9CFB51E38649ED9C2AD4EBDAD2AE42, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=BF54167049CC4061B97304731A9EC218, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:37,108 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:37,111 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533056912_0001_1_00_000000_0
2016-05-18 00:57:37,112 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533056912_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:37,112 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533056912_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:37,118 INFO input.MRInput (MRInput.java:initialize(440)) - BD9CFB51E38649ED9C2AD4EBDAD2AE42 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:37,118 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: BD9CFB51E38649ED9C2AD4EBDAD2AE42
2016-05-18 00:57:37,118 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:37,124 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D823A53DDF454BA79F82290F84B0DC16, ordinal: 0
2016-05-18 00:57:37,124 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D823A53DDF454BA79F82290F84B0DC16, mem on start (mb), free: 196, total: 350, max: 672
2016-05-18 00:57:37,124 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:37,124 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:37,124 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:37,124 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:37,124 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BD9CFB51E38649ED9C2AD4EBDAD2AE42:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [BF54167049CC4061B97304731A9EC218:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:37,125 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:37,125 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:37,125 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:37,125 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BD9CFB51E38649ED9C2AD4EBDAD2AE42:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [BF54167049CC4061B97304731A9EC218:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:37,125 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:37,125 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BD9CFB51E38649ED9C2AD4EBDAD2AE42 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:37,126 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:37,126 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:37,126 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BD9CFB51E38649ED9C2AD4EBDAD2AE42
2016-05-18 00:57:37,128 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:37,129 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533056912_0001_1_00_000000_0
2016-05-18 00:57:37,132 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:37,132 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:37,132 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - BD9CFB51E38649ED9C2AD4EBDAD2AE42 initialized RecordReader from event
2016-05-18 00:57:37,133 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: BD9CFB51E38649ED9C2AD4EBDAD2AE42
2016-05-18 00:57:37,133 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: CBEF9904F6BC489FB1154FFF9FB1541C
2016-05-18 00:57:37,133 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] CBEF9904F6BC489FB1154FFF9FB1541C
2016-05-18 00:57:37,135 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:37,138 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:37,138 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - BF54167049CC4061B97304731A9EC218 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@441fb02d, comparator=cascading.tuple.hadoop.util.TupleComparator@7c677d4b, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:37,138 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for BF54167049CC4061B97304731A9EC218: , UsingHashComparator=false
2016-05-18 00:57:37,163 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:37,163 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:37,163 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for BF54167049CC4061B97304731A9EC218: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:37,163 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - BF54167049CC4061B97304731A9EC218: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:37,164 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:37,164 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D823A53DDF454BA79F82290F84B0DC16, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:37,167 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533056912_0001_1_00_000000_0
2016-05-18 00:57:37,167 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D823A53DDF454BA79F82290F84B0DC16, mem on close (mb), free: 213, total: 364, max: 672
2016-05-18 00:57:37,168 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - BF54167049CC4061B97304731A9EC218: Starting flush of map output
2016-05-18 00:57:37,168 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - BF54167049CC4061B97304731A9EC218: Span0.length = 20, perItem = 6
2016-05-18 00:57:37,168 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - BF54167049CC4061B97304731A9EC218: done sorting span=0, length=20, time=0
2016-05-18 00:57:37,168 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - BF54167049CC4061B97304731A9EC218: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:37,170 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - BF54167049CC4061B97304731A9EC218: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001_wd/localmode-local-dir/output/attempt_1463533056912_0001_1_00_000000_0_10129_0/file.out
2016-05-18 00:57:37,174 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:37 Completed running task attempt: attempt_1463533056912_0001_1_00_000000_0
2016-05-18 00:57:37,175 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533056912_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:37,175 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533056912_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:37,176 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533056912_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533057164, Process_Duration=3, Process_End_Time=1463533057167, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:37,176 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533056912_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:37,177 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:37,177 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D823A53DDF454BA79F82290F84B0DC16, taskAttemptId=attempt_1463533056912_0001_1_00_000000_0, creationTime=1463533057103, allocationTime=1463533057104, startTime=1463533057106, finishTime=1463533057176, timeTaken=70, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:37,177 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:37,177 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533056912_0001_1_00_000000_0
2016-05-18 00:57:37,177 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,177 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:TASK_FINISHED]: vertexName=D823A53DDF454BA79F82290F84B0DC16, taskId=task_1463533056912_0001_1_00_000000, startTime=1463533057106, finishTime=1463533057177, timeTaken=71, status=SUCCEEDED, successfulAttemptID=attempt_1463533056912_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533057164, Process_Duration=3, Process_End_Time=1463533057167, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:37,178 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533056912_0001_00_000001
2016-05-18 00:57:37,179 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:37,179 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:VERTEX_FINISHED]: vertexName=D823A53DDF454BA79F82290F84B0DC16, vertexId=vertex_1463533056912_0001_1_00, initRequestedTime=1463533057083, initedTime=1463533057096, startRequestedTime=1463533057084, startedTime=1463533057096, finishTime=1463533057179, timeTaken=83, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533057164, Process_Duration=3, Process_End_Time=1463533057167, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533057106, firstTasksToStart=[ task_1463533056912_0001_1_00_000000 ], lastTaskFinishTime=1463533057176, lastTasksToFinish=[ task_1463533056912_0001_1_00_000000 ], minTaskDuration=70, maxTaskDuration=70, avgTaskDuration=70.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533056912_0001_1_00_000000 ], longestDurationTasks=[ task_1463533056912_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:37,180 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:37,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533056912_0001_00_000001, stoppedTime=1463533057180, exitStatus=0
2016-05-18 00:57:37,180 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218] attempt: attempt_1463533056912_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:37,181 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:37,181 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:37,181 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: BF54167049CC4061B97304731A9EC218
2016-05-18 00:57:37,181 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533056912_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:37,181 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D823A53DDF454BA79F82290F84B0DC16, destinationVertexName=BF54167049CC4061B97304731A9EC218. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:37,181 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]
2016-05-18 00:57:37,182 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:TASK_STARTED]: vertexName=BF54167049CC4061B97304731A9EC218, taskId=task_1463533056912_0001_1_01_000000, scheduledTime=1463533057182, launchTime=1463533057182
2016-05-18 00:57:37,182 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]
2016-05-18 00:57:37,184 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533056912_0001_00_000002, launchTime=1463533057184
2016-05-18 00:57:37,185 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533056912_0001_00_000002
2016-05-18 00:57:37,186 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533056912_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533056912_0001_1_01_000000_0
2016-05-18 00:57:37,186 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533056912_0001_1_01_000000_0] started. Is using containerId: [container_1463533056912_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:37,187 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BF54167049CC4061B97304731A9EC218, taskAttemptId=attempt_1463533056912_0001_1_01_000000_0, startTime=1463533057186, containerId=container_1463533056912_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:37 Starting to run new task attempt: attempt_1463533056912_0001_1_01_000000_0
2016-05-18 00:57:37,187 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533056912_0001_1_01_000000_0
2016-05-18 00:57:37,187 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,187 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:37,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E] every/(1/1) ...latform/0_'field'_RESULTS, VertexName: BF54167049CC4061B97304731A9EC218, VertexParallelism: 1, TaskAttemptID:attempt_1463533056912_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D823A53DDF454BA79F82290F84B0DC16, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=A318480F527B4CC0986FF8C27EE9DCA2, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:37,187 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:37,188 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533056912_0001_1_01_000000_0
2016-05-18 00:57:37,189 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533056912_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:37,189 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533056912_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]
2016-05-18 00:57:37,189 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533056912_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:37,189 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533056912_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:37,190 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533056912_0001_00_000001, Exiting
2016-05-18 00:57:37,190 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533056912_0001_00_000001
2016-05-18 00:57:37,190 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533056912_0001_00_000001
2016-05-18 00:57:37,194 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:37,194 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533056912_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:37,196 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533056912_0001_00_000001 completed successfully
2016-05-18 00:57:37,202 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:37,202 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BF54167049CC4061B97304731A9EC218, ordinal: 1
2016-05-18 00:57:37,202 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BF54167049CC4061B97304731A9EC218, mem on start (mb), free: 206, total: 364, max: 672
2016-05-18 00:57:37,202 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:37,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:37,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:37,203 INFO output.MROutput (MROutput.java:initialize(421)) - A318480F527B4CC0986FF8C27EE9DCA2: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:37,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:37,204 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A318480F527B4CC0986FF8C27EE9DCA2:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [D823A53DDF454BA79F82290F84B0DC16:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:37,204 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:37,204 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:37,204 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:37,205 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A318480F527B4CC0986FF8C27EE9DCA2:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [D823A53DDF454BA79F82290F84B0DC16:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:37,205 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:37,205 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D823A53DDF454BA79F82290F84B0DC16 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:37,205 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:37,205 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:37,207 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D823A53DDF454BA79F82290F84B0DC16: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:37,207 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D823A53DDF454BA79F82290F84B0DC16: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:37,209 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D823A53DDF454BA79F82290F84B0DC16 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:37,209 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D823A53DDF454BA79F82290F84B0DC16
2016-05-18 00:57:37,210 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:37,210 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533056912_0001_1_01_000000_0
2016-05-18 00:57:37,212 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D823A53DDF454BA79F82290F84B0DC16: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:37,214 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533056912_0001_1_00_000000_0_10129} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533057214, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:37,214 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D823A53DDF454BA79F82290F84B0DC16
2016-05-18 00:57:37,214 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:37,215 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:37,215 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D823A53DDF454BA79F82290F84B0DC16, wasInterrupted=false
2016-05-18 00:57:37,215 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: CBEF9904F6BC489FB1154FFF9FB1541C
2016-05-18 00:57:37,219 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:37,219 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_RESULTS"], id: A318480F527B4CC0986FF8C27EE9DCA2
2016-05-18 00:57:37,219 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_RESULTS"]
2016-05-18 00:57:37,220 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - D823A53DDF454BA79F82290F84B0DC16: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:37,220 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] CBEF9904F6BC489FB1154FFF9FB1541C, for 1 inputs
2016-05-18 00:57:37,221 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:37,222 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:37,222 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001_wd/localmode-local-dir/output/attempt_1463533056912_0001_1_00_000000_0_10129/file.out, len=181, isLocal=true
2016-05-18 00:57:37,222 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:37,223 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:37,223 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:37,223 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:37,223 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D823A53DDF454BA79F82290F84B0DC16
2016-05-18 00:57:37,223 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D823A53DDF454BA79F82290F84B0DC16: Shuffle Runner thread complete
2016-05-18 00:57:37,224 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BF54167049CC4061B97304731A9EC218, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:37,225 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:37,225 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D823A53DDF454BA79F82290F84B0DC16: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:37,230 INFO output.MROutput (MROutput.java:close(526)) - A318480F527B4CC0986FF8C27EE9DCA2 closed
2016-05-18 00:57:37,230 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533056912_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:37,231 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533056912_0001_1_01_000000_0
2016-05-18 00:57:37,231 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BF54167049CC4061B97304731A9EC218, mem on close (mb), free: 202, total: 364, max: 672
2016-05-18 00:57:37,231 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D823A53DDF454BA79F82290F84B0DC16
2016-05-18 00:57:37,231 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D823A53DDF454BA79F82290F84B0DC16: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:37,231 INFO output.MROutput (MROutput.java:close(526)) - A318480F527B4CC0986FF8C27EE9DCA2 closed
2016-05-18 00:57:37,232 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:37 Completed running task attempt: attempt_1463533056912_0001_1_01_000000_0
2016-05-18 00:57:37,233 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533056912_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:37,233 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533056912_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:37,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533056912_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533057224, Process_Duration=7, Process_End_Time=1463533057231, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:37,234 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533056912_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:37,234 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:37,234 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BF54167049CC4061B97304731A9EC218, taskAttemptId=attempt_1463533056912_0001_1_01_000000_0, creationTime=1463533057182, allocationTime=1463533057183, startTime=1463533057186, finishTime=1463533057234, timeTaken=48, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:37,235 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533056912_0001_1_01_000000_0
2016-05-18 00:57:37,235 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:TASK_FINISHED]: vertexName=BF54167049CC4061B97304731A9EC218, taskId=task_1463533056912_0001_1_01_000000, startTime=1463533057186, finishTime=1463533057235, timeTaken=49, status=SUCCEEDED, successfulAttemptID=attempt_1463533056912_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533057224, Process_Duration=7, Process_End_Time=1463533057231, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:37,235 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:37,236 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,236 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:37,236 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218]
2016-05-18 00:57:37,237 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:VERTEX_FINISHED]: vertexName=BF54167049CC4061B97304731A9EC218, vertexId=vertex_1463533056912_0001_1_01, initRequestedTime=1463533057084, initedTime=1463533057088, startRequestedTime=1463533057097, startedTime=1463533057097, finishTime=1463533057236, timeTaken=139, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533057224, Process_Duration=7, Process_End_Time=1463533057231, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533057186, firstTasksToStart=[ task_1463533056912_0001_1_01_000000 ], lastTaskFinishTime=1463533057234, lastTasksToFinish=[ task_1463533056912_0001_1_01_000000 ], minTaskDuration=48, maxTaskDuration=48, avgTaskDuration=48.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533056912_0001_1_01_000000 ], longestDurationTasks=[ task_1463533056912_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:37,237 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:37,237 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533056912_0001_00_000002, stoppedTime=1463533057237, exitStatus=0
2016-05-18 00:57:37,237 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:37,237 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:37,238 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533056912_0001_1_00 [D823A53DDF454BA79F82290F84B0DC16]
2016-05-18 00:57:37,238 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533056912_0001_1
2016-05-18 00:57:37,238 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533056912_0001_1
2016-05-18 00:57:37,238 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533056912_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:37,238 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: A318480F527B4CC0986FF8C27EE9DCA2 for vertex: vertex_1463533056912_0001_1_01 [BF54167049CC4061B97304731A9EC218], outputName: A318480F527B4CC0986FF8C27EE9DCA2
2016-05-18 00:57:37,238 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533056912_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:37,239 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533056912_0001_00_000002
2016-05-18 00:57:37,239 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533056912_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:37,239 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533056912_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:37,239 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533056912_0001_00_000002, Exiting
2016-05-18 00:57:37,240 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533056912_0001_00_000002
2016-05-18 00:57:37,240 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533056912_0001_00_000002
2016-05-18 00:57:37,240 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533056912_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:37,240 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533056912_0001_00_000002 completed successfully
2016-05-18 00:57:37,246 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:A318480F527B4CC0986FF8C27EE9DCA2 of vertex/vertexGroup:BF54167049CC4061B97304731A9EC218 isVertexGroupOutput:false
2016-05-18 00:57:37,246 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:37,248 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533056912_0001_1, queueSize=0
2016-05-18 00:57:37,248 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533056912_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533056912_0001_1, startTime=1463533057083, finishTime=1463533057246, timeTaken=163, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=410, AM_GC_TIME_MILLIS=10, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=16, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066114388, Process_Duration=10, Process_End_Time=2927066114398, Read_Duration=2, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:37,248 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533056912_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:37,248 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533056912_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:37 Completed Dag: dag_1463533056912_0001_1
2016-05-18 00:57:37,249 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533056912_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:37,249 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533056912_0001_1_post
2016-05-18 00:57:37,249 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,249 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:37,249 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:37,249 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E] every/(1/1) ...latform/0_'field'_RESULTS, with id=dag_1463533056912_0001_1
2016-05-18 00:57:37,249 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E] every/(1/1) ...latform/0_'field'_RESULTS, with id=dag_1463533056912_0001_1
2016-05-18 00:57:37,264 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...latform/0_'field'_RESULTS, applicationId=application_1463533056912_0001
2016-05-18 00:57:37,266 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:37,266 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:37,266 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:37,266 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:37,266 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:37,267 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:37,267 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:37,268 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:37,268 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:37,269 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:37,269 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:37,269 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:37,269 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:37,271 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:37,275 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B947409479BD4961BCF80E636549000B/F3B0327149134E19B0E46642247CBA8E/.tez/application_1463533056912_0001
2016-05-18 00:57:37,275 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_RESULTS/_temporary
2016-05-18 00:57:37,276 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.392
2016-05-18 00:57:37,344 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_'field'_RESULTS
2016-05-18 00:57:37,368 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:37,368 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:37,404 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 00:57:37,417 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.020
2016-05-18 00:57:37,417 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:37,418 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:37,418 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:37,432 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:37,437 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_ALL
2016-05-18 00:57:37,468 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:37,468 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:37,485 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:37,486 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:37,486 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_ALL"]
2016-05-18 00:57:37,486 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:37,486 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:37,486 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:37,488 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...blyplatform/0_'field'_ALL
2016-05-18 00:57:37,488 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:37,501 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:37,506 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:37,506 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:37,506 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:37,515 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001 doesn't exist and is created
2016-05-18 00:57:37,529 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:37,529 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:37,531 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001_wd
2016-05-18 00:57:37,541 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533057506_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:37,541 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:37,541 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:37,542 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:37,543 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:37,543 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:37,543 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:37,543 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:37,545 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:37,545 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:37,545 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:37,545 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:37,545 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:37,545 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:37,545 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:37,546 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:37,546 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:37,546 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:37,546 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:37,546 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:37,547 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:37,548 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:37,548 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:37,548 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001_wd/localmode-log-dir
2016-05-18 00:57:37,549 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001_wd/localmode-log-dir/history.txt.appattempt_1463533057506_0001_000000, maxErrors=10
2016-05-18 00:57:37,549 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:37,549 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533057506_0001_000000, appSubmitTime=1463533057540, launchTime=1463533057541
2016-05-18 00:57:37,552 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:37,556 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:36303
2016-05-18 00:57:37,557 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533057506
2016-05-18 00:57:37,558 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533057506_0001_000000, startTime=1463533057558
2016-05-18 00:57:37,558 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:37,629 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:37,630 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:37,630 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...blyplatform/0_'field'_ALL, applicationId=application_1463533057506_0001, dagName=[F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7] every/(1/1) ...blyplatform/0_'field'_ALL
2016-05-18 00:57:37,645 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7] every/(1/1) ...blyplatform/0_'field'_ALL
2016-05-18 00:57:37,645 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:37,646 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533057506_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001_wd/localmode-log-dir/dag_1463533057506_0001_1.dot
2016-05-18 00:57:37,647 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001_wd/localmode-log-dir/dag_1463533057506_0001_1-tez-dag.pb.txt
2016-05-18 00:57:37,653 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533057506_0001_1
2016-05-18 00:57:37,653 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,653 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7] every/(1/1) ...blyplatform/0_'field'_ALL
2016-05-18 00:57:37 Running Dag: dag_1463533057506_0001_1
2016-05-18 00:57:37,662 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533057506_0001_1, submitTime=1463533057645
2016-05-18 00:57:37,663 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001/tez-conf.pb" } size: 36913 timestamp: 1463533057000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:37,663 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,664 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:37,665 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001/tez-conf.pb" } size: 36913 timestamp: 1463533057000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:37,665 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]
2016-05-18 00:57:37,665 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:37,665 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:37,667 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533057506_0001_1, initTime=1463533057667
2016-05-18 00:57:37,667 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533057506_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:37,667 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:37,668 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:DAG_STARTED]: dagID=dag_1463533057506_0001_1, startTime=1463533057668
2016-05-18 00:57:37,668 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533057506_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:37,668 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D4F27DCFA6B3438EB5C0D4D47BF03A4B : {2804F0048CD94382904F9A983F5044D1={InputName=2804F0048CD94382904F9A983F5044D1}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:37,668 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 2804F0048CD94382904F9A983F5044D1, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:37,668 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,669 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,669 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,669 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,669 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...blyplatform/0_'field'_ALL, applicationId=application_1463533057506_0001, dagName=[F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7] every/(1/1) ...blyplatform/0_'field'_ALL
2016-05-18 00:57:37,669 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:37,669 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]
2016-05-18 00:57:37,671 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 2804F0048CD94382904F9A983F5044D1 on vertex vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,681 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:37,681 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]
2016-05-18 00:57:37,681 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]
2016-05-18 00:57:37,682 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533057506_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:37,682 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3], numAdditionalOutputs=1
2016-05-18 00:57:37,683 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=1179C3914EAD4FEAB916A3E02CFBC652, vertex=vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:37,685 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 2804F0048CD94382904F9A983F5044D1 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:37,685 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D5DBDE01B7F34469A09DA8DDE155C4A3:1179C3914EAD4FEAB916A3E02CFBC652 using oldmapred API
2016-05-18 00:57:37,685 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=1179C3914EAD4FEAB916A3E02CFBC652, vertexName=D5DBDE01B7F34469A09DA8DDE155C4A3, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:37,687 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D5DBDE01B7F34469A09DA8DDE155C4A3, vertexId=vertex_1463533057506_0001_1_01, initRequestedTime=1463533057669, initedTime=1463533057682, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:37,687 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:37,687 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:37,687 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:37,688 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:37,689 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:37,689 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 2804F0048CD94382904F9A983F5044D1 on vertex vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,691 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:37,691 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {2804F0048CD94382904F9A983F5044D1=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:37,692 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] parallelism set to 1
2016-05-18 00:57:37,692 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:37,693 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D4F27DCFA6B3438EB5C0D4D47BF03A4B, vertexId=vertex_1463533057506_0001_1_00, initRequestedTime=1463533057668, initedTime=1463533057693, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:37,693 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] with distanceFromRoot: 0
2016-05-18 00:57:37,693 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:37,694 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533057506_0001_1_00, startRequestedTime=1463533057669, startedTime=1463533057694
2016-05-18 00:57:37,694 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533057506_0001, with dag id: dag_1463533057506_0001_1
2016-05-18 00:57:37,694 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533057506_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:37,695 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:37,695 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D4F27DCFA6B3438EB5C0D4D47BF03A4B
2016-05-18 00:57:37,695 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,696 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533057506_0001_1_00 for vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3] numStartedSources: 1 numSources: 1
2016-05-18 00:57:37,696 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3] with distanceFromRoot: 1
2016-05-18 00:57:37,696 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533057506_0001_1_01, startRequestedTime=1463533057696, startedTime=1463533057696
2016-05-18 00:57:37,696 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:37,697 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:TASK_STARTED]: vertexName=D4F27DCFA6B3438EB5C0D4D47BF03A4B, taskId=task_1463533057506_0001_1_00_000000, scheduledTime=1463533057697, launchTime=1463533057697
2016-05-18 00:57:37,697 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,699 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:37,700 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D5DBDE01B7F34469A09DA8DDE155C4A3 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:37,702 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D4F27DCFA6B3438EB5C0D4D47BF03A4B in vertex: D5DBDE01B7F34469A09DA8DDE155C4A3 numBipartiteSourceTasks: 1
2016-05-18 00:57:37,704 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533057506_0001_00_000001, launchTime=1463533057703
2016-05-18 00:57:37,705 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533057506_0001_00_000001
2016-05-18 00:57:37,705 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533057506_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533057506_0001_1_00_000000_0
2016-05-18 00:57:37,706 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533057506_0001_1_00_000000_0] started. Is using containerId: [container_1463533057506_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:37,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D4F27DCFA6B3438EB5C0D4D47BF03A4B, taskAttemptId=attempt_1463533057506_0001_1_00_000000_0, startTime=1463533057705, containerId=container_1463533057506_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:37 Starting to run new task attempt: attempt_1463533057506_0001_1_00_000000_0
2016-05-18 00:57:37,706 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533057506_0001_1_00_000000_0
2016-05-18 00:57:37,706 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,707 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:37,707 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7] every/(1/1) ...blyplatform/0_'field'_ALL, VertexName: D4F27DCFA6B3438EB5C0D4D47BF03A4B, VertexParallelism: 1, TaskAttemptID:attempt_1463533057506_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=2804F0048CD94382904F9A983F5044D1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D5DBDE01B7F34469A09DA8DDE155C4A3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:37,708 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:37,708 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533057506_0001_1_00_000000_0
2016-05-18 00:57:37,709 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533057506_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:37,709 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533057506_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:37,711 INFO input.MRInput (MRInput.java:initialize(440)) - 2804F0048CD94382904F9A983F5044D1 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:37,711 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 2804F0048CD94382904F9A983F5044D1
2016-05-18 00:57:37,712 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:37,718 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D4F27DCFA6B3438EB5C0D4D47BF03A4B, ordinal: 0
2016-05-18 00:57:37,718 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4F27DCFA6B3438EB5C0D4D47BF03A4B, mem on start (mb), free: 301, total: 363, max: 672
2016-05-18 00:57:37,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:37,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:37,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:37,719 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:37,719 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[2804F0048CD94382904F9A983F5044D1:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D5DBDE01B7F34469A09DA8DDE155C4A3:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:37,719 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:37,720 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:37,720 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:37,720 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[2804F0048CD94382904F9A983F5044D1:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D5DBDE01B7F34469A09DA8DDE155C4A3:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:37,720 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:37,721 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 2804F0048CD94382904F9A983F5044D1 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:37,721 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 2804F0048CD94382904F9A983F5044D1
2016-05-18 00:57:37,721 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:37,721 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:37,722 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:37,722 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533057506_0001_1_00_000000_0
2016-05-18 00:57:37,723 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:37,723 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 2804F0048CD94382904F9A983F5044D1 initialized RecordReader from event
2016-05-18 00:57:37,725 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:37,726 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 2804F0048CD94382904F9A983F5044D1
2016-05-18 00:57:37,726 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 112F12E344774CD39AEA330B9F2FB014
2016-05-18 00:57:37,726 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 112F12E344774CD39AEA330B9F2FB014
2016-05-18 00:57:37,727 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:37,731 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:37,731 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D5DBDE01B7F34469A09DA8DDE155C4A3 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@56bc7b43, comparator=cascading.tuple.hadoop.util.TupleComparator@70409831, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:37,731 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D5DBDE01B7F34469A09DA8DDE155C4A3: , UsingHashComparator=false
2016-05-18 00:57:37,745 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:37,745 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:37,746 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D5DBDE01B7F34469A09DA8DDE155C4A3: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:37,746 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D5DBDE01B7F34469A09DA8DDE155C4A3: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:37,746 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:37,746 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D4F27DCFA6B3438EB5C0D4D47BF03A4B, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:37,749 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533057506_0001_1_00_000000_0
2016-05-18 00:57:37,749 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D4F27DCFA6B3438EB5C0D4D47BF03A4B, mem on close (mb), free: 199, total: 363, max: 672
2016-05-18 00:57:37,749 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D5DBDE01B7F34469A09DA8DDE155C4A3: Starting flush of map output
2016-05-18 00:57:37,750 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D5DBDE01B7F34469A09DA8DDE155C4A3: Span0.length = 20, perItem = 6
2016-05-18 00:57:37,750 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D5DBDE01B7F34469A09DA8DDE155C4A3: done sorting span=0, length=20, time=0
2016-05-18 00:57:37,750 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D5DBDE01B7F34469A09DA8DDE155C4A3: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:37,751 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D5DBDE01B7F34469A09DA8DDE155C4A3: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001_wd/localmode-local-dir/output/attempt_1463533057506_0001_1_00_000000_0_10135_0/file.out
2016-05-18 00:57:37,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:37 Completed running task attempt: attempt_1463533057506_0001_1_00_000000_0
2016-05-18 00:57:37,757 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533057506_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:37,757 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533057506_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:37,757 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533057506_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533057746, Process_Duration=3, Process_End_Time=1463533057749, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:37,758 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533057506_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:37,758 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:37,758 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D4F27DCFA6B3438EB5C0D4D47BF03A4B, taskAttemptId=attempt_1463533057506_0001_1_00_000000_0, creationTime=1463533057697, allocationTime=1463533057700, startTime=1463533057705, finishTime=1463533057758, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:37,758 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533057506_0001_1_00_000000_0
2016-05-18 00:57:37,759 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:TASK_FINISHED]: vertexName=D4F27DCFA6B3438EB5C0D4D47BF03A4B, taskId=task_1463533057506_0001_1_00_000000, startTime=1463533057705, finishTime=1463533057758, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463533057506_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533057746, Process_Duration=3, Process_End_Time=1463533057749, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:37,758 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:37,759 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,759 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:37,759 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,760 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:VERTEX_FINISHED]: vertexName=D4F27DCFA6B3438EB5C0D4D47BF03A4B, vertexId=vertex_1463533057506_0001_1_00, initRequestedTime=1463533057668, initedTime=1463533057693, startRequestedTime=1463533057669, startedTime=1463533057694, finishTime=1463533057759, timeTaken=65, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533057746, Process_Duration=3, Process_End_Time=1463533057749, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533057705, firstTasksToStart=[ task_1463533057506_0001_1_00_000000 ], lastTaskFinishTime=1463533057758, lastTasksToFinish=[ task_1463533057506_0001_1_00_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533057506_0001_1_00_000000 ], longestDurationTasks=[ task_1463533057506_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:37,760 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:37,760 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3] attempt: attempt_1463533057506_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:37,760 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:37,760 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:37,759 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533057506_0001_00_000001
2016-05-18 00:57:37,761 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533057506_0001_00_000001, stoppedTime=1463533057761, exitStatus=0
2016-05-18 00:57:37,760 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D5DBDE01B7F34469A09DA8DDE155C4A3
2016-05-18 00:57:37,761 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533057506_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:37,761 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D4F27DCFA6B3438EB5C0D4D47BF03A4B, destinationVertexName=D5DBDE01B7F34469A09DA8DDE155C4A3. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:37,761 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]
2016-05-18 00:57:37,761 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533057506_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:37,762 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533057506_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:37,762 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:TASK_STARTED]: vertexName=D5DBDE01B7F34469A09DA8DDE155C4A3, taskId=task_1463533057506_0001_1_01_000000, scheduledTime=1463533057761, launchTime=1463533057761
2016-05-18 00:57:37,762 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533057506_0001_00_000001, Exiting
2016-05-18 00:57:37,762 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533057506_0001_00_000001
2016-05-18 00:57:37,762 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533057506_0001_00_000001
2016-05-18 00:57:37,762 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]
2016-05-18 00:57:37,763 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533057506_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:37,764 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533057506_0001_00_000001 completed successfully
2016-05-18 00:57:37,764 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533057506_0001_00_000002, launchTime=1463533057764
2016-05-18 00:57:37,765 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533057506_0001_00_000002
2016-05-18 00:57:37,765 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533057506_0001_1_01_000000_0] started. Is using containerId: [container_1463533057506_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:37,765 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D5DBDE01B7F34469A09DA8DDE155C4A3, taskAttemptId=attempt_1463533057506_0001_1_01_000000_0, startTime=1463533057765, containerId=container_1463533057506_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:37,766 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533057506_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533057506_0001_1_01_000000_0
2016-05-18 00:57:37 Starting to run new task attempt: attempt_1463533057506_0001_1_01_000000_0
2016-05-18 00:57:37,767 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533057506_0001_1_01_000000_0
2016-05-18 00:57:37,767 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,768 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:37,768 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7] every/(1/1) ...blyplatform/0_'field'_ALL, VertexName: D5DBDE01B7F34469A09DA8DDE155C4A3, VertexParallelism: 1, TaskAttemptID:attempt_1463533057506_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D4F27DCFA6B3438EB5C0D4D47BF03A4B, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=1179C3914EAD4FEAB916A3E02CFBC652, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:37,769 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:37,769 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533057506_0001_1_01_000000_0
2016-05-18 00:57:37,771 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533057506_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]
2016-05-18 00:57:37,771 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533057506_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:37,772 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:37,775 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:37,779 INFO output.MROutput (MROutput.java:initialize(421)) - 1179C3914EAD4FEAB916A3E02CFBC652: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:37,779 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D5DBDE01B7F34469A09DA8DDE155C4A3, ordinal: 1
2016-05-18 00:57:37,779 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D5DBDE01B7F34469A09DA8DDE155C4A3, mem on start (mb), free: 194, total: 363, max: 672
2016-05-18 00:57:37,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:37,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:37,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:37,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:37,780 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[1179C3914EAD4FEAB916A3E02CFBC652:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [D4F27DCFA6B3438EB5C0D4D47BF03A4B:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:37,781 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:37,781 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:37,781 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:37,781 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[1179C3914EAD4FEAB916A3E02CFBC652:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [D4F27DCFA6B3438EB5C0D4D47BF03A4B:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:37,782 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:37,782 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D4F27DCFA6B3438EB5C0D4D47BF03A4B being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:37,782 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D4F27DCFA6B3438EB5C0D4D47BF03A4B: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:37,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:37,783 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:37,783 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D4F27DCFA6B3438EB5C0D4D47BF03A4B: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:37,784 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D4F27DCFA6B3438EB5C0D4D47BF03A4B with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:37,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D4F27DCFA6B3438EB5C0D4D47BF03A4B
2016-05-18 00:57:37,785 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:37,786 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533057506_0001_1_01_000000_0
2016-05-18 00:57:37,789 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:37,789 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D4F27DCFA6B3438EB5C0D4D47BF03A4B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:37,789 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 112F12E344774CD39AEA330B9F2FB014
2016-05-18 00:57:37,789 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_ALL"], id: 1179C3914EAD4FEAB916A3E02CFBC652
2016-05-18 00:57:37,790 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_ALL"]
2016-05-18 00:57:37,790 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 112F12E344774CD39AEA330B9F2FB014, for 1 inputs
2016-05-18 00:57:37,791 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533057506_0001_1_00_000000_0_10135} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533057791, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:37,791 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D4F27DCFA6B3438EB5C0D4D47BF03A4B
2016-05-18 00:57:37,792 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:37,792 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D4F27DCFA6B3438EB5C0D4D47BF03A4B, wasInterrupted=false
2016-05-18 00:57:37,792 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - D4F27DCFA6B3438EB5C0D4D47BF03A4B: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:37,793 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:37,793 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:37,794 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:37,794 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001_wd/localmode-local-dir/output/attempt_1463533057506_0001_1_00_000000_0_10135/file.out, len=181, isLocal=true
2016-05-18 00:57:37,795 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:37,795 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:37,795 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:37,795 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:37,795 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D4F27DCFA6B3438EB5C0D4D47BF03A4B
2016-05-18 00:57:37,795 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D5DBDE01B7F34469A09DA8DDE155C4A3, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:37,795 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D4F27DCFA6B3438EB5C0D4D47BF03A4B: Shuffle Runner thread complete
2016-05-18 00:57:37,796 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:37,796 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D4F27DCFA6B3438EB5C0D4D47BF03A4B: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:37,802 INFO output.MROutput (MROutput.java:close(526)) - 1179C3914EAD4FEAB916A3E02CFBC652 closed
2016-05-18 00:57:37,802 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533057506_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:37,802 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533057506_0001_1_01_000000_0
2016-05-18 00:57:37,802 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D5DBDE01B7F34469A09DA8DDE155C4A3, mem on close (mb), free: 191, total: 363, max: 672
2016-05-18 00:57:37,803 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D4F27DCFA6B3438EB5C0D4D47BF03A4B
2016-05-18 00:57:37,803 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D4F27DCFA6B3438EB5C0D4D47BF03A4B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:37,803 INFO output.MROutput (MROutput.java:close(526)) - 1179C3914EAD4FEAB916A3E02CFBC652 closed
2016-05-18 00:57:37,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:37 Completed running task attempt: attempt_1463533057506_0001_1_01_000000_0
2016-05-18 00:57:37,803 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533057506_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:37,803 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533057506_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:37,804 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533057506_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533057795, Process_Duration=7, Process_End_Time=1463533057802, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:37,804 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533057506_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:37,805 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:37,805 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D5DBDE01B7F34469A09DA8DDE155C4A3, taskAttemptId=attempt_1463533057506_0001_1_01_000000_0, creationTime=1463533057762, allocationTime=1463533057763, startTime=1463533057765, finishTime=1463533057804, timeTaken=39, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:37,805 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:37,805 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,805 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533057506_0001_00_000002
2016-05-18 00:57:37,805 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533057506_0001_1_01_000000_0
2016-05-18 00:57:37,805 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:TASK_FINISHED]: vertexName=D5DBDE01B7F34469A09DA8DDE155C4A3, taskId=task_1463533057506_0001_1_01_000000, startTime=1463533057765, finishTime=1463533057805, timeTaken=40, status=SUCCEEDED, successfulAttemptID=attempt_1463533057506_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533057795, Process_Duration=7, Process_End_Time=1463533057802, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:37,805 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:37,805 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3]
2016-05-18 00:57:37,806 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:VERTEX_FINISHED]: vertexName=D5DBDE01B7F34469A09DA8DDE155C4A3, vertexId=vertex_1463533057506_0001_1_01, initRequestedTime=1463533057669, initedTime=1463533057682, startRequestedTime=1463533057696, startedTime=1463533057696, finishTime=1463533057806, timeTaken=110, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533057795, Process_Duration=7, Process_End_Time=1463533057802, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533057765, firstTasksToStart=[ task_1463533057506_0001_1_01_000000 ], lastTaskFinishTime=1463533057804, lastTasksToFinish=[ task_1463533057506_0001_1_01_000000 ], minTaskDuration=39, maxTaskDuration=39, avgTaskDuration=39.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533057506_0001_1_01_000000 ], longestDurationTasks=[ task_1463533057506_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:37,806 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:37,807 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533057506_0001_00_000002, stoppedTime=1463533057806, exitStatus=0
2016-05-18 00:57:37,807 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:37,807 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:37,807 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533057506_0001_1_00 [D4F27DCFA6B3438EB5C0D4D47BF03A4B]
2016-05-18 00:57:37,807 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533057506_0001_1
2016-05-18 00:57:37,807 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533057506_0001_1
2016-05-18 00:57:37,807 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533057506_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:37,807 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 1179C3914EAD4FEAB916A3E02CFBC652 for vertex: vertex_1463533057506_0001_1_01 [D5DBDE01B7F34469A09DA8DDE155C4A3], outputName: 1179C3914EAD4FEAB916A3E02CFBC652
2016-05-18 00:57:37,807 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533057506_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:37,813 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:1179C3914EAD4FEAB916A3E02CFBC652 of vertex/vertexGroup:D5DBDE01B7F34469A09DA8DDE155C4A3 isVertexGroupOutput:false
2016-05-18 00:57:37,813 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:37,815 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533057506_0001_1, queueSize=0
2016-05-18 00:57:37,816 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533057506_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533057506_0001_1, startTime=1463533057667, finishTime=1463533057813, timeTaken=146, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=230, AM_GC_TIME_MILLIS=8, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066115541, Process_Duration=10, Process_End_Time=2927066115551, Read_Duration=1, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:37,816 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533057506_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:37,816 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533057506_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:37,817 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533057506_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:37 Completed Dag: dag_1463533057506_0001_1
2016-05-18 00:57:37,817 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533057506_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:37,817 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533057506_0001_00_000002, Exiting
2016-05-18 00:57:37,817 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533057506_0001_00_000002
2016-05-18 00:57:37,818 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533057506_0001_00_000002
2016-05-18 00:57:37,818 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533057506_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:37,818 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533057506_0001_00_000002 completed successfully
2016-05-18 00:57:37,817 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533057506_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:37,818 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533057506_0001_1_post
2016-05-18 00:57:37,818 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:37,819 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:37,819 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:37,819 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7] every/(1/1) ...blyplatform/0_'field'_ALL, with id=dag_1463533057506_0001_1
2016-05-18 00:57:37,820 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7] every/(1/1) ...blyplatform/0_'field'_ALL, with id=dag_1463533057506_0001_1
2016-05-18 00:57:37,823 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...blyplatform/0_'field'_ALL, applicationId=application_1463533057506_0001
2016-05-18 00:57:37,825 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:37,825 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:37,825 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:37,826 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:37,826 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:37,827 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:37,827 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:37,828 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:37,828 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:37,829 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:37,829 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:37,829 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:37,829 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:37,830 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:37,835 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F6EC53A5A8DA4AEBA7E297E7E1B4CA42/B3DC7F4214D94B7A9654605631127FA7/.tez/application_1463533057506_0001
2016-05-18 00:57:37,835 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_ALL/_temporary
2016-05-18 00:57:37,836 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.350
2016-05-18 00:57:37,907 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_'field'_ALL
2016-05-18 00:57:37,939 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:37,940 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:37,969 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.004
2016-05-18 00:57:37,971 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.006
2016-05-18 00:57:37,972 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:37,972 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:37,973 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_'field'_REPLACE
2016-05-18 00:57:37,995 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:37,995 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:37,998 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:37,998 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:38,036 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 00:57:38,043 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.018
2016-05-18 00:57:38,043 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:38,043 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:38,044 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:38,058 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:38,064 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_SWAP
2016-05-18 00:57:38,109 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:38,110 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:38,110 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_SWAP"]
2016-05-18 00:57:38,111 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:38,111 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:38,111 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:38,113 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/0_'field'_SWAP
2016-05-18 00:57:38,113 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:38,133 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:38,138 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:38,138 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:38,138 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:38,146 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001 doesn't exist and is created
2016-05-18 00:57:38,160 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:38,160 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:38,161 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001_wd
2016-05-18 00:57:38,170 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533058138_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:38,171 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:38,171 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:38,172 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:38,172 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:38,173 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:38,173 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:38,173 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:38,175 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:38,175 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:38,175 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:38,175 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:38,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:38,176 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:38,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:38,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:38,176 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:38,177 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:38,177 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:38,177 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:38,179 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:38,179 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:38,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:38,180 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001_wd/localmode-log-dir
2016-05-18 00:57:38,180 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001_wd/localmode-log-dir/history.txt.appattempt_1463533058138_0001_000000, maxErrors=10
2016-05-18 00:57:38,181 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:38,181 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533058138_0001_000000, appSubmitTime=1463533058170, launchTime=1463533058170
2016-05-18 00:57:38,183 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:38,188 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:35805
2016-05-18 00:57:38,189 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533058138
2016-05-18 00:57:38,191 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533058138_0001_000000, startTime=1463533058191
2016-05-18 00:57:38,191 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:38,261 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:38,261 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:38,261 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/0_'field'_SWAP, applicationId=application_1463533058138_0001, dagName=[95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE] every/(1/1) ...lyplatform/0_'field'_SWAP
2016-05-18 00:57:38,277 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE] every/(1/1) ...lyplatform/0_'field'_SWAP
2016-05-18 00:57:38,278 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:38,278 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533058138_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001_wd/localmode-log-dir/dag_1463533058138_0001_1.dot
2016-05-18 00:57:38,279 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001_wd/localmode-log-dir/dag_1463533058138_0001_1-tez-dag.pb.txt
2016-05-18 00:57:38,285 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533058138_0001_1
2016-05-18 00:57:38,285 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,286 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE] every/(1/1) ...lyplatform/0_'field'_SWAP
2016-05-18 00:57:38 Running Dag: dag_1463533058138_0001_1
2016-05-18 00:57:38,294 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533058138_0001_1, submitTime=1463533058278
2016-05-18 00:57:38,295 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001/tez-conf.pb" } size: 36913 timestamp: 1463533058000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:38,296 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,296 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:38,298 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001/tez-conf.pb" } size: 36913 timestamp: 1463533058000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:38,298 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]
2016-05-18 00:57:38,298 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:38,298 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:38,299 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533058138_0001_1, initTime=1463533058299
2016-05-18 00:57:38,299 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533058138_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:38,300 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:38,300 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:DAG_STARTED]: dagID=dag_1463533058138_0001_1, startTime=1463533058300
2016-05-18 00:57:38,300 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533058138_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:38,300 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/0_'field'_SWAP, applicationId=application_1463533058138_0001, dagName=[95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE] every/(1/1) ...lyplatform/0_'field'_SWAP
2016-05-18 00:57:38,300 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B8AB1BBA98034B869604F62BA8F57D44 : {9B113D3457B0432D9380C61926E940E0={InputName=9B113D3457B0432D9380C61926E940E0}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:38,300 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 9B113D3457B0432D9380C61926E940E0, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:38,301 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,301 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,301 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,301 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,301 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:38,302 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]
2016-05-18 00:57:38,302 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 9B113D3457B0432D9380C61926E940E0 on vertex vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,304 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 9B113D3457B0432D9380C61926E940E0 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:38,305 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:38,305 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]
2016-05-18 00:57:38,305 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]
2016-05-18 00:57:38,305 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533058138_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:38,306 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3], numAdditionalOutputs=1
2016-05-18 00:57:38,306 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=4A3618F4F82E4CB0AD39504EEF10BF8B, vertex=vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:38,306 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:38,307 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:38,307 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:38,307 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:38,307 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 9B113D3457B0432D9380C61926E940E0 on vertex vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,308 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E61C1CAD37CE4058A95DA8751FCD01D3:4A3618F4F82E4CB0AD39504EEF10BF8B using oldmapred API
2016-05-18 00:57:38,308 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=4A3618F4F82E4CB0AD39504EEF10BF8B, vertexName=E61C1CAD37CE4058A95DA8751FCD01D3, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:38,309 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E61C1CAD37CE4058A95DA8751FCD01D3, vertexId=vertex_1463533058138_0001_1_01, initRequestedTime=1463533058302, initedTime=1463533058306, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:38,309 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:38,310 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:38,310 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {9B113D3457B0432D9380C61926E940E0=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:38,310 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] parallelism set to 1
2016-05-18 00:57:38,310 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:38,311 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B8AB1BBA98034B869604F62BA8F57D44, vertexId=vertex_1463533058138_0001_1_00, initRequestedTime=1463533058300, initedTime=1463533058311, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:38,311 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] with distanceFromRoot: 0
2016-05-18 00:57:38,311 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:38,312 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533058138_0001_1_00, startRequestedTime=1463533058302, startedTime=1463533058311
2016-05-18 00:57:38,312 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533058138_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:38,312 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:38,312 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533058138_0001_1_00 for vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3] numStartedSources: 1 numSources: 1
2016-05-18 00:57:38,312 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3] with distanceFromRoot: 1
2016-05-18 00:57:38,313 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B8AB1BBA98034B869604F62BA8F57D44
2016-05-18 00:57:38,313 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,313 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533058138_0001_1_01, startRequestedTime=1463533058312, startedTime=1463533058313
2016-05-18 00:57:38,315 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:38,315 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:TASK_STARTED]: vertexName=B8AB1BBA98034B869604F62BA8F57D44, taskId=task_1463533058138_0001_1_00_000000, scheduledTime=1463533058315, launchTime=1463533058315
2016-05-18 00:57:38,316 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,318 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E61C1CAD37CE4058A95DA8751FCD01D3 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:38,318 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:38,319 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B8AB1BBA98034B869604F62BA8F57D44 in vertex: E61C1CAD37CE4058A95DA8751FCD01D3 numBipartiteSourceTasks: 1
2016-05-18 00:57:38,320 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533058138_0001_00_000001, launchTime=1463533058320
2016-05-18 00:57:38,321 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533058138_0001_00_000001
2016-05-18 00:57:38,321 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533058138_0001, with dag id: dag_1463533058138_0001_1
2016-05-18 00:57:38,321 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533058138_0001_1_00_000000_0] started. Is using containerId: [container_1463533058138_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:38,321 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B8AB1BBA98034B869604F62BA8F57D44, taskAttemptId=attempt_1463533058138_0001_1_00_000000_0, startTime=1463533058321, containerId=container_1463533058138_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:38,322 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533058138_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533058138_0001_1_00_000000_0
2016-05-18 00:57:38 Starting to run new task attempt: attempt_1463533058138_0001_1_00_000000_0
2016-05-18 00:57:38,322 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533058138_0001_1_00_000000_0
2016-05-18 00:57:38,322 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,323 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:38,323 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE] every/(1/1) ...lyplatform/0_'field'_SWAP, VertexName: B8AB1BBA98034B869604F62BA8F57D44, VertexParallelism: 1, TaskAttemptID:attempt_1463533058138_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9B113D3457B0432D9380C61926E940E0, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E61C1CAD37CE4058A95DA8751FCD01D3, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:38,323 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:38,324 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533058138_0001_1_00_000000_0
2016-05-18 00:57:38,324 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533058138_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:38,325 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533058138_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:38,327 INFO input.MRInput (MRInput.java:initialize(440)) - 9B113D3457B0432D9380C61926E940E0 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:38,327 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 9B113D3457B0432D9380C61926E940E0
2016-05-18 00:57:38,327 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:38,335 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B8AB1BBA98034B869604F62BA8F57D44, ordinal: 0
2016-05-18 00:57:38,335 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8AB1BBA98034B869604F62BA8F57D44, mem on start (mb), free: 274, total: 364, max: 672
2016-05-18 00:57:38,335 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:38,335 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:38,335 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:38,335 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:38,335 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[9B113D3457B0432D9380C61926E940E0:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E61C1CAD37CE4058A95DA8751FCD01D3:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:38,335 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:38,336 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:38,336 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:38,336 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[9B113D3457B0432D9380C61926E940E0:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E61C1CAD37CE4058A95DA8751FCD01D3:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:38,336 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:38,336 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 9B113D3457B0432D9380C61926E940E0 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:38,336 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:38,336 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:38,336 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 9B113D3457B0432D9380C61926E940E0
2016-05-18 00:57:38,337 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:38,337 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533058138_0001_1_00_000000_0
2016-05-18 00:57:38,338 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:38,338 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 9B113D3457B0432D9380C61926E940E0 initialized RecordReader from event
2016-05-18 00:57:38,340 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:38,341 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 9B113D3457B0432D9380C61926E940E0
2016-05-18 00:57:38,341 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: BC531B062B7F4160AEA401C8FB875513
2016-05-18 00:57:38,342 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] BC531B062B7F4160AEA401C8FB875513
2016-05-18 00:57:38,344 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:38,347 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:38,347 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E61C1CAD37CE4058A95DA8751FCD01D3 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@2868220c, comparator=cascading.tuple.hadoop.util.TupleComparator@4c93568d, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:38,348 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E61C1CAD37CE4058A95DA8751FCD01D3: , UsingHashComparator=false
2016-05-18 00:57:38,361 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:38,361 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:38,361 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E61C1CAD37CE4058A95DA8751FCD01D3: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:38,362 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E61C1CAD37CE4058A95DA8751FCD01D3: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:38,362 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:38,362 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B8AB1BBA98034B869604F62BA8F57D44, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:38,365 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533058138_0001_1_00_000000_0
2016-05-18 00:57:38,365 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8AB1BBA98034B869604F62BA8F57D44, mem on close (mb), free: 171, total: 364, max: 672
2016-05-18 00:57:38,365 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E61C1CAD37CE4058A95DA8751FCD01D3: Starting flush of map output
2016-05-18 00:57:38,365 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E61C1CAD37CE4058A95DA8751FCD01D3: Span0.length = 20, perItem = 6
2016-05-18 00:57:38,365 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E61C1CAD37CE4058A95DA8751FCD01D3: done sorting span=0, length=20, time=0
2016-05-18 00:57:38,365 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E61C1CAD37CE4058A95DA8751FCD01D3: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:38,367 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E61C1CAD37CE4058A95DA8751FCD01D3: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001_wd/localmode-local-dir/output/attempt_1463533058138_0001_1_00_000000_0_10141_0/file.out
2016-05-18 00:57:38,372 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:38 Completed running task attempt: attempt_1463533058138_0001_1_00_000000_0
2016-05-18 00:57:38,372 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533058138_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:38,373 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533058138_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:38,373 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533058138_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533058362, Process_Duration=3, Process_End_Time=1463533058365, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:38,373 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533058138_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:38,374 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:38,374 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:38,374 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B8AB1BBA98034B869604F62BA8F57D44, taskAttemptId=attempt_1463533058138_0001_1_00_000000_0, creationTime=1463533058316, allocationTime=1463533058320, startTime=1463533058321, finishTime=1463533058373, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:38,374 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533058138_0001_1_00_000000_0
2016-05-18 00:57:38,375 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,375 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:TASK_FINISHED]: vertexName=B8AB1BBA98034B869604F62BA8F57D44, taskId=task_1463533058138_0001_1_00_000000, startTime=1463533058321, finishTime=1463533058374, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463533058138_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533058362, Process_Duration=3, Process_End_Time=1463533058365, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:38,376 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:38,376 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,377 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:VERTEX_FINISHED]: vertexName=B8AB1BBA98034B869604F62BA8F57D44, vertexId=vertex_1463533058138_0001_1_00, initRequestedTime=1463533058300, initedTime=1463533058311, startRequestedTime=1463533058302, startedTime=1463533058311, finishTime=1463533058376, timeTaken=65, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533058362, Process_Duration=3, Process_End_Time=1463533058365, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533058321, firstTasksToStart=[ task_1463533058138_0001_1_00_000000 ], lastTaskFinishTime=1463533058373, lastTasksToFinish=[ task_1463533058138_0001_1_00_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533058138_0001_1_00_000000 ], longestDurationTasks=[ task_1463533058138_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:38,377 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:38,377 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533058138_0001_00_000001, stoppedTime=1463533058377, exitStatus=0
2016-05-18 00:57:38,377 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3] attempt: attempt_1463533058138_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:38,377 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:38,377 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:38,378 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533058138_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:38,375 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533058138_0001_00_000001
2016-05-18 00:57:38,377 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E61C1CAD37CE4058A95DA8751FCD01D3
2016-05-18 00:57:38,378 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B8AB1BBA98034B869604F62BA8F57D44, destinationVertexName=E61C1CAD37CE4058A95DA8751FCD01D3. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:38,379 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]
2016-05-18 00:57:38,379 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:TASK_STARTED]: vertexName=E61C1CAD37CE4058A95DA8751FCD01D3, taskId=task_1463533058138_0001_1_01_000000, scheduledTime=1463533058379, launchTime=1463533058379
2016-05-18 00:57:38,379 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]
2016-05-18 00:57:38,381 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533058138_0001_00_000002, launchTime=1463533058381
2016-05-18 00:57:38,379 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533058138_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:38,382 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533058138_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:38,383 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533058138_0001_00_000001, Exiting
2016-05-18 00:57:38,383 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533058138_0001_00_000001
2016-05-18 00:57:38,383 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533058138_0001_00_000001
2016-05-18 00:57:38,383 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533058138_0001_00_000002
2016-05-18 00:57:38,384 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533058138_0001_1_01_000000_0] started. Is using containerId: [container_1463533058138_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:38,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E61C1CAD37CE4058A95DA8751FCD01D3, taskAttemptId=attempt_1463533058138_0001_1_01_000000_0, startTime=1463533058384, containerId=container_1463533058138_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:38,385 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533058138_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533058138_0001_1_01_000000_0
2016-05-18 00:57:38 Starting to run new task attempt: attempt_1463533058138_0001_1_01_000000_0
2016-05-18 00:57:38,385 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533058138_0001_1_01_000000_0
2016-05-18 00:57:38,385 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,385 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:38,385 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE] every/(1/1) ...lyplatform/0_'field'_SWAP, VertexName: E61C1CAD37CE4058A95DA8751FCD01D3, VertexParallelism: 1, TaskAttemptID:attempt_1463533058138_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B8AB1BBA98034B869604F62BA8F57D44, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=4A3618F4F82E4CB0AD39504EEF10BF8B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:38,386 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:38,386 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533058138_0001_1_01_000000_0
2016-05-18 00:57:38,386 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533058138_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]
2016-05-18 00:57:38,387 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533058138_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:38,387 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533058138_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:38,387 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533058138_0001_00_000001 completed successfully
2016-05-18 00:57:38,389 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:38,391 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:38,399 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E61C1CAD37CE4058A95DA8751FCD01D3, ordinal: 1
2016-05-18 00:57:38,399 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E61C1CAD37CE4058A95DA8751FCD01D3, mem on start (mb), free: 167, total: 364, max: 672
2016-05-18 00:57:38,399 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:38,400 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:38,400 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:38,404 INFO output.MROutput (MROutput.java:initialize(421)) - 4A3618F4F82E4CB0AD39504EEF10BF8B: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:38,404 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:38,405 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B8AB1BBA98034B869604F62BA8F57D44:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [4A3618F4F82E4CB0AD39504EEF10BF8B:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:38,405 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:38,405 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:38,405 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:38,405 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B8AB1BBA98034B869604F62BA8F57D44:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [4A3618F4F82E4CB0AD39504EEF10BF8B:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:38,405 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:38,405 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B8AB1BBA98034B869604F62BA8F57D44 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:38,406 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B8AB1BBA98034B869604F62BA8F57D44: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:38,407 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B8AB1BBA98034B869604F62BA8F57D44: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:38,407 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:38,408 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:38,408 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B8AB1BBA98034B869604F62BA8F57D44 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:38,411 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B8AB1BBA98034B869604F62BA8F57D44
2016-05-18 00:57:38,411 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:38,415 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533058138_0001_1_01_000000_0
2016-05-18 00:57:38,419 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:38,419 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8AB1BBA98034B869604F62BA8F57D44: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:38,420 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: BC531B062B7F4160AEA401C8FB875513
2016-05-18 00:57:38,420 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_SWAP"], id: 4A3618F4F82E4CB0AD39504EEF10BF8B
2016-05-18 00:57:38,421 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_SWAP"]
2016-05-18 00:57:38,421 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533058138_0001_1_00_000000_0_10141} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533058421, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:38,421 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] BC531B062B7F4160AEA401C8FB875513, for 1 inputs
2016-05-18 00:57:38,421 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B8AB1BBA98034B869604F62BA8F57D44
2016-05-18 00:57:38,422 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:38,422 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B8AB1BBA98034B869604F62BA8F57D44, wasInterrupted=false
2016-05-18 00:57:38,423 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - B8AB1BBA98034B869604F62BA8F57D44: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:38,423 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:38,424 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:38,425 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:38,425 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001_wd/localmode-local-dir/output/attempt_1463533058138_0001_1_00_000000_0_10141/file.out, len=181, isLocal=true
2016-05-18 00:57:38,425 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:38,425 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:38,425 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:38,426 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:38,426 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B8AB1BBA98034B869604F62BA8F57D44
2016-05-18 00:57:38,426 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B8AB1BBA98034B869604F62BA8F57D44: Shuffle Runner thread complete
2016-05-18 00:57:38,426 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E61C1CAD37CE4058A95DA8751FCD01D3, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:38,427 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:38,428 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B8AB1BBA98034B869604F62BA8F57D44: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:38,433 INFO output.MROutput (MROutput.java:close(526)) - 4A3618F4F82E4CB0AD39504EEF10BF8B closed
2016-05-18 00:57:38,433 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533058138_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:38,433 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533058138_0001_1_01_000000_0
2016-05-18 00:57:38,433 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E61C1CAD37CE4058A95DA8751FCD01D3, mem on close (mb), free: 163, total: 364, max: 672
2016-05-18 00:57:38,434 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B8AB1BBA98034B869604F62BA8F57D44
2016-05-18 00:57:38,434 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8AB1BBA98034B869604F62BA8F57D44: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:38,434 INFO output.MROutput (MROutput.java:close(526)) - 4A3618F4F82E4CB0AD39504EEF10BF8B closed
2016-05-18 00:57:38,434 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:38 Completed running task attempt: attempt_1463533058138_0001_1_01_000000_0
2016-05-18 00:57:38,434 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533058138_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:38,434 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533058138_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:38,435 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533058138_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533058426, Process_Duration=7, Process_End_Time=1463533058433, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:38,436 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533058138_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:38,436 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E61C1CAD37CE4058A95DA8751FCD01D3, taskAttemptId=attempt_1463533058138_0001_1_01_000000_0, creationTime=1463533058379, allocationTime=1463533058380, startTime=1463533058384, finishTime=1463533058436, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:38,436 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:38,436 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:38,436 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,436 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533058138_0001_1_01_000000_0
2016-05-18 00:57:38,437 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:TASK_FINISHED]: vertexName=E61C1CAD37CE4058A95DA8751FCD01D3, taskId=task_1463533058138_0001_1_01_000000, startTime=1463533058384, finishTime=1463533058436, timeTaken=52, status=SUCCEEDED, successfulAttemptID=attempt_1463533058138_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533058426, Process_Duration=7, Process_End_Time=1463533058433, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:38,437 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:38,437 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3]
2016-05-18 00:57:38,438 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:VERTEX_FINISHED]: vertexName=E61C1CAD37CE4058A95DA8751FCD01D3, vertexId=vertex_1463533058138_0001_1_01, initRequestedTime=1463533058302, initedTime=1463533058306, startRequestedTime=1463533058312, startedTime=1463533058313, finishTime=1463533058437, timeTaken=124, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533058426, Process_Duration=7, Process_End_Time=1463533058433, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533058384, firstTasksToStart=[ task_1463533058138_0001_1_01_000000 ], lastTaskFinishTime=1463533058436, lastTasksToFinish=[ task_1463533058138_0001_1_01_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533058138_0001_1_01_000000 ], longestDurationTasks=[ task_1463533058138_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:38,438 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:38,438 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533058138_0001_00_000002
2016-05-18 00:57:38,438 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533058138_0001_00_000002, stoppedTime=1463533058438, exitStatus=0
2016-05-18 00:57:38,438 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:38,438 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533058138_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:38,438 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533058138_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:38,439 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533058138_0001_00_000002, Exiting
2016-05-18 00:57:38,439 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533058138_0001_00_000002
2016-05-18 00:57:38,439 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533058138_0001_00_000002
2016-05-18 00:57:38,438 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:38,439 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533058138_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:38,439 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533058138_0001_00_000002 completed successfully
2016-05-18 00:57:38,439 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533058138_0001_1_00 [B8AB1BBA98034B869604F62BA8F57D44]
2016-05-18 00:57:38,439 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533058138_0001_1
2016-05-18 00:57:38,440 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533058138_0001_1
2016-05-18 00:57:38,440 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 4A3618F4F82E4CB0AD39504EEF10BF8B for vertex: vertex_1463533058138_0001_1_01 [E61C1CAD37CE4058A95DA8751FCD01D3], outputName: 4A3618F4F82E4CB0AD39504EEF10BF8B
2016-05-18 00:57:38,441 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533058138_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:38,442 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533058138_0001_00_000002
2016-05-18 00:57:38,446 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:4A3618F4F82E4CB0AD39504EEF10BF8B of vertex/vertexGroup:E61C1CAD37CE4058A95DA8751FCD01D3 isVertexGroupOutput:false
2016-05-18 00:57:38,446 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:38,448 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533058138_0001_1, queueSize=0
2016-05-18 00:57:38,449 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058138_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533058138_0001_1, startTime=1463533058300, finishTime=1463533058446, timeTaken=146, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=310, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=15, MERGE_PHASE_TIME=19, FIRST_EVENT_RECEIVED=12, LAST_EVENT_RECEIVED=12, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066116788, Process_Duration=10, Process_End_Time=2927066116798, Read_Duration=1, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:38,449 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533058138_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:38,449 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533058138_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:38 Completed Dag: dag_1463533058138_0001_1
2016-05-18 00:57:38,449 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533058138_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:38,449 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533058138_0001_1_post
2016-05-18 00:57:38,449 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,450 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:38,450 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:38,450 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE] every/(1/1) ...lyplatform/0_'field'_SWAP, with id=dag_1463533058138_0001_1
2016-05-18 00:57:38,450 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE] every/(1/1) ...lyplatform/0_'field'_SWAP, with id=dag_1463533058138_0001_1
2016-05-18 00:57:38,455 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/0_'field'_SWAP, applicationId=application_1463533058138_0001
2016-05-18 00:57:38,456 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:38,456 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:38,457 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:38,457 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:38,457 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:38,457 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:38,457 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:38,457 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:38,458 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:38,458 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:38,458 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:38,458 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:38,458 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:38,459 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:38,463 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/95A449A5A32F44898F6BBF8BAEF4F467/F22B429FDB714EFB9F64DA3C39AC4CDE/.tez/application_1463533058138_0001
2016-05-18 00:57:38,463 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_'field'_SWAP/_temporary
2016-05-18 00:57:38,464 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.354
2016-05-18 00:57:38,532 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_'field'_SWAP
2016-05-18 00:57:38,554 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:38,554 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:38,571 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:38,572 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:38,590 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 00:57:38,593 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:38,593 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:38,593 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:38,593 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:38,610 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:38,615 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_-1
2016-05-18 00:57:38,670 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:38,671 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:38,671 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_-1"]
2016-05-18 00:57:38,671 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:38,671 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:38,671 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:38,672 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...mblyplatform/0_UNKNOWN_-1
2016-05-18 00:57:38,672 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:38,685 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:38,689 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:38,689 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:38,689 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:38,698 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001 doesn't exist and is created
2016-05-18 00:57:38,713 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:38,713 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:38,714 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001_wd
2016-05-18 00:57:38,723 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533058689_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:38,723 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:38,723 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:38,724 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:38,724 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:38,724 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:38,724 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:38,725 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:38,726 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:38,726 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:38,726 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:38,727 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:38,727 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:38,727 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:38,727 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:38,727 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:38,727 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:38,727 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:38,728 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:38,728 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:38,729 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:38,729 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:38,730 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:38,730 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001_wd/localmode-log-dir
2016-05-18 00:57:38,730 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001_wd/localmode-log-dir/history.txt.appattempt_1463533058689_0001_000000, maxErrors=10
2016-05-18 00:57:38,730 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:38,730 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533058689_0001_000000, appSubmitTime=1463533058722, launchTime=1463533058722
2016-05-18 00:57:38,733 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:38,735 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:51261
2016-05-18 00:57:38,735 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533058689
2016-05-18 00:57:38,737 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533058689_0001_000000, startTime=1463533058737
2016-05-18 00:57:38,738 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:38,813 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:38,813 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:38,814 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...mblyplatform/0_UNKNOWN_-1, applicationId=application_1463533058689_0001, dagName=[0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E] every/(1/1) ...mblyplatform/0_UNKNOWN_-1
2016-05-18 00:57:38,829 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E] every/(1/1) ...mblyplatform/0_UNKNOWN_-1
2016-05-18 00:57:38,829 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:38,830 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533058689_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001_wd/localmode-log-dir/dag_1463533058689_0001_1.dot
2016-05-18 00:57:38,831 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001_wd/localmode-log-dir/dag_1463533058689_0001_1-tez-dag.pb.txt
2016-05-18 00:57:38,838 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533058689_0001_1
2016-05-18 00:57:38,838 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,838 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E] every/(1/1) ...mblyplatform/0_UNKNOWN_-1
2016-05-18 00:57:38 Running Dag: dag_1463533058689_0001_1
2016-05-18 00:57:38,846 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533058689_0001_1, submitTime=1463533058829
2016-05-18 00:57:38,847 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001/tez-conf.pb" } size: 36913 timestamp: 1463533058000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:38,848 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,848 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:38,848 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001/tez-conf.pb" } size: 36913 timestamp: 1463533058000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:38,848 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]
2016-05-18 00:57:38,848 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:38,849 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:38,849 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533058689_0001_1, initTime=1463533058849
2016-05-18 00:57:38,849 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533058689_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:38,849 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:38,850 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:DAG_STARTED]: dagID=dag_1463533058689_0001_1, startTime=1463533058850
2016-05-18 00:57:38,850 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533058689_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:38,850 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A211CF8A9E55474097741C67742D40FF : {571A14425CF848A79407AC9F9E765692={InputName=571A14425CF848A79407AC9F9E765692}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:38,850 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 571A14425CF848A79407AC9F9E765692, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:38,850 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,850 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,851 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,850 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...mblyplatform/0_UNKNOWN_-1, applicationId=application_1463533058689_0001, dagName=[0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E] every/(1/1) ...mblyplatform/0_UNKNOWN_-1
2016-05-18 00:57:38,851 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,851 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:38,852 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]
2016-05-18 00:57:38,855 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 571A14425CF848A79407AC9F9E765692 on vertex vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,855 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:38,855 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]
2016-05-18 00:57:38,855 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]
2016-05-18 00:57:38,856 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533058689_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:38,856 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262], numAdditionalOutputs=1
2016-05-18 00:57:38,856 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=562ED65A1AB745E2A886331788491D78, vertex=vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:38,856 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 571A14425CF848A79407AC9F9E765692 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:38,858 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:38,858 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for B0D6D82DBC514BC5B868F2860E8A6262:562ED65A1AB745E2A886331788491D78 using oldmapred API
2016-05-18 00:57:38,858 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=562ED65A1AB745E2A886331788491D78, vertexName=B0D6D82DBC514BC5B868F2860E8A6262, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:38,858 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:38,858 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:38,859 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:38,859 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B0D6D82DBC514BC5B868F2860E8A6262, vertexId=vertex_1463533058689_0001_1_01, initRequestedTime=1463533058852, initedTime=1463533058856, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:38,859 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:38,859 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 571A14425CF848A79407AC9F9E765692 on vertex vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,860 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:38,860 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {571A14425CF848A79407AC9F9E765692=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:38,861 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] parallelism set to 1
2016-05-18 00:57:38,861 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:38,862 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A211CF8A9E55474097741C67742D40FF, vertexId=vertex_1463533058689_0001_1_00, initRequestedTime=1463533058850, initedTime=1463533058861, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:38,862 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] with distanceFromRoot: 0
2016-05-18 00:57:38,862 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:38,863 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533058689_0001_1_00, startRequestedTime=1463533058852, startedTime=1463533058862
2016-05-18 00:57:38,863 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533058689_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:38,863 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:38,864 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A211CF8A9E55474097741C67742D40FF
2016-05-18 00:57:38,864 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,864 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533058689_0001_1_00 for vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262] numStartedSources: 1 numSources: 1
2016-05-18 00:57:38,864 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262] with distanceFromRoot: 1
2016-05-18 00:57:38,865 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533058689_0001_1_01, startRequestedTime=1463533058864, startedTime=1463533058865
2016-05-18 00:57:38,866 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:38,866 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: B0D6D82DBC514BC5B868F2860E8A6262 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:38,866 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:TASK_STARTED]: vertexName=A211CF8A9E55474097741C67742D40FF, taskId=task_1463533058689_0001_1_00_000000, scheduledTime=1463533058866, launchTime=1463533058866
2016-05-18 00:57:38,867 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A211CF8A9E55474097741C67742D40FF in vertex: B0D6D82DBC514BC5B868F2860E8A6262 numBipartiteSourceTasks: 1
2016-05-18 00:57:38,867 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,870 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:38,872 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533058689_0001_00_000001, launchTime=1463533058872
2016-05-18 00:57:38,873 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533058689_0001_00_000001
2016-05-18 00:57:38,873 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533058689_0001_1_00_000000_0] started. Is using containerId: [container_1463533058689_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:38,874 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A211CF8A9E55474097741C67742D40FF, taskAttemptId=attempt_1463533058689_0001_1_00_000000_0, startTime=1463533058873, containerId=container_1463533058689_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:38,874 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533058689_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533058689_0001_1_00_000000_0
2016-05-18 00:57:38 Starting to run new task attempt: attempt_1463533058689_0001_1_00_000000_0
2016-05-18 00:57:38,875 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533058689_0001_1_00_000000_0
2016-05-18 00:57:38,876 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,876 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:38,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E] every/(1/1) ...mblyplatform/0_UNKNOWN_-1, VertexName: A211CF8A9E55474097741C67742D40FF, VertexParallelism: 1, TaskAttemptID:attempt_1463533058689_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=571A14425CF848A79407AC9F9E765692, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B0D6D82DBC514BC5B868F2860E8A6262, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:38,876 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:38,877 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533058689_0001, with dag id: dag_1463533058689_0001_1
2016-05-18 00:57:38,877 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533058689_0001_1_00_000000_0
2016-05-18 00:57:38,880 INFO input.MRInput (MRInput.java:initialize(440)) - 571A14425CF848A79407AC9F9E765692 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:38,880 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:38,881 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533058689_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:38,881 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533058689_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:38,880 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 571A14425CF848A79407AC9F9E765692
2016-05-18 00:57:38,888 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A211CF8A9E55474097741C67742D40FF, ordinal: 0
2016-05-18 00:57:38,888 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A211CF8A9E55474097741C67742D40FF, mem on start (mb), free: 262, total: 364, max: 672
2016-05-18 00:57:38,889 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:38,889 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:38,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:38,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:38,890 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[571A14425CF848A79407AC9F9E765692:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [B0D6D82DBC514BC5B868F2860E8A6262:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:38,890 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:38,890 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:38,890 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:38,890 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[571A14425CF848A79407AC9F9E765692:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [B0D6D82DBC514BC5B868F2860E8A6262:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:38,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:38,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 571A14425CF848A79407AC9F9E765692 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:38,891 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:38,891 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:38,891 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 571A14425CF848A79407AC9F9E765692
2016-05-18 00:57:38,891 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:38,892 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533058689_0001_1_00_000000_0
2016-05-18 00:57:38,895 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:38,896 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 571A14425CF848A79407AC9F9E765692
2016-05-18 00:57:38,896 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: C092818E59E749A2BAB612A7CD7DABA5
2016-05-18 00:57:38,896 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] C092818E59E749A2BAB612A7CD7DABA5
2016-05-18 00:57:38,897 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:38,898 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:38,899 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 571A14425CF848A79407AC9F9E765692 initialized RecordReader from event
2016-05-18 00:57:38,901 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:38,901 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - B0D6D82DBC514BC5B868F2860E8A6262 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@d4415b6, comparator=cascading.tuple.hadoop.util.TupleComparator@4c6cc8cf, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:38,901 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for B0D6D82DBC514BC5B868F2860E8A6262: , UsingHashComparator=false
2016-05-18 00:57:38,914 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:38,914 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:38,915 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for B0D6D82DBC514BC5B868F2860E8A6262: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:38,915 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - B0D6D82DBC514BC5B868F2860E8A6262: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:38,915 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:38,915 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A211CF8A9E55474097741C67742D40FF, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:38,918 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533058689_0001_1_00_000000_0
2016-05-18 00:57:38,918 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A211CF8A9E55474097741C67742D40FF, mem on close (mb), free: 160, total: 364, max: 672
2016-05-18 00:57:38,918 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - B0D6D82DBC514BC5B868F2860E8A6262: Starting flush of map output
2016-05-18 00:57:38,919 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - B0D6D82DBC514BC5B868F2860E8A6262: Span0.length = 20, perItem = 6
2016-05-18 00:57:38,919 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - B0D6D82DBC514BC5B868F2860E8A6262: done sorting span=0, length=20, time=0
2016-05-18 00:57:38,919 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - B0D6D82DBC514BC5B868F2860E8A6262: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:38,921 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - B0D6D82DBC514BC5B868F2860E8A6262: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001_wd/localmode-local-dir/output/attempt_1463533058689_0001_1_00_000000_0_10147_0/file.out
2016-05-18 00:57:38,924 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:38 Completed running task attempt: attempt_1463533058689_0001_1_00_000000_0
2016-05-18 00:57:38,926 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533058689_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:38,926 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533058689_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:38,926 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533058689_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533058915, Process_Duration=3, Process_End_Time=1463533058918, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:38,927 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533058689_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:38,927 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:38,927 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A211CF8A9E55474097741C67742D40FF, taskAttemptId=attempt_1463533058689_0001_1_00_000000_0, creationTime=1463533058867, allocationTime=1463533058872, startTime=1463533058873, finishTime=1463533058927, timeTaken=54, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:38,928 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:38,928 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533058689_0001_1_00_000000_0
2016-05-18 00:57:38,928 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,928 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:TASK_FINISHED]: vertexName=A211CF8A9E55474097741C67742D40FF, taskId=task_1463533058689_0001_1_00_000000, startTime=1463533058873, finishTime=1463533058928, timeTaken=55, status=SUCCEEDED, successfulAttemptID=attempt_1463533058689_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533058915, Process_Duration=3, Process_End_Time=1463533058918, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:38,929 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533058689_0001_00_000001
2016-05-18 00:57:38,929 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:38,929 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:38,930 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:VERTEX_FINISHED]: vertexName=A211CF8A9E55474097741C67742D40FF, vertexId=vertex_1463533058689_0001_1_00, initRequestedTime=1463533058850, initedTime=1463533058861, startRequestedTime=1463533058852, startedTime=1463533058862, finishTime=1463533058929, timeTaken=67, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533058915, Process_Duration=3, Process_End_Time=1463533058918, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533058873, firstTasksToStart=[ task_1463533058689_0001_1_00_000000 ], lastTaskFinishTime=1463533058927, lastTasksToFinish=[ task_1463533058689_0001_1_00_000000 ], minTaskDuration=54, maxTaskDuration=54, avgTaskDuration=54.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533058689_0001_1_00_000000 ], longestDurationTasks=[ task_1463533058689_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:38,930 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:38,930 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262] attempt: attempt_1463533058689_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:38,930 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533058689_0001_00_000001, stoppedTime=1463533058930, exitStatus=0
2016-05-18 00:57:38,930 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B0D6D82DBC514BC5B868F2860E8A6262
2016-05-18 00:57:38,931 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A211CF8A9E55474097741C67742D40FF, destinationVertexName=B0D6D82DBC514BC5B868F2860E8A6262. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:38,931 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]
2016-05-18 00:57:38,930 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:38,931 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:38,932 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533058689_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:38,932 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:TASK_STARTED]: vertexName=B0D6D82DBC514BC5B868F2860E8A6262, taskId=task_1463533058689_0001_1_01_000000, scheduledTime=1463533058932, launchTime=1463533058932
2016-05-18 00:57:38,932 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]
2016-05-18 00:57:38,934 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533058689_0001_00_000002, launchTime=1463533058934
2016-05-18 00:57:38,935 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533058689_0001_00_000002
2016-05-18 00:57:38,936 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533058689_0001_1_01_000000_0] started. Is using containerId: [container_1463533058689_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:38,936 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B0D6D82DBC514BC5B868F2860E8A6262, taskAttemptId=attempt_1463533058689_0001_1_01_000000_0, startTime=1463533058935, containerId=container_1463533058689_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:38,936 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533058689_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533058689_0001_1_01_000000_0
2016-05-18 00:57:38 Starting to run new task attempt: attempt_1463533058689_0001_1_01_000000_0
2016-05-18 00:57:38,938 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533058689_0001_1_01_000000_0
2016-05-18 00:57:38,938 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:38,938 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:38,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E] every/(1/1) ...mblyplatform/0_UNKNOWN_-1, VertexName: B0D6D82DBC514BC5B868F2860E8A6262, VertexParallelism: 1, TaskAttemptID:attempt_1463533058689_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A211CF8A9E55474097741C67742D40FF, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=562ED65A1AB745E2A886331788491D78, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:38,938 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:38,939 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533058689_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:38,939 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533058689_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:38,939 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533058689_0001_00_000001, Exiting
2016-05-18 00:57:38,940 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533058689_0001_00_000001
2016-05-18 00:57:38,940 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533058689_0001_00_000001
2016-05-18 00:57:38,941 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533058689_0001_1_01_000000_0
2016-05-18 00:57:38,942 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533058689_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]
2016-05-18 00:57:38,942 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533058689_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:38,943 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533058689_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:38,943 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533058689_0001_00_000001 completed successfully
2016-05-18 00:57:38,945 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:38,949 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:38,956 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B0D6D82DBC514BC5B868F2860E8A6262, ordinal: 1
2016-05-18 00:57:38,957 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B0D6D82DBC514BC5B868F2860E8A6262, mem on start (mb), free: 155, total: 364, max: 672
2016-05-18 00:57:38,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:38,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:38,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:38,963 INFO output.MROutput (MROutput.java:initialize(421)) - 562ED65A1AB745E2A886331788491D78: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:38,964 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:38,964 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A211CF8A9E55474097741C67742D40FF:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [562ED65A1AB745E2A886331788491D78:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:38,964 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:38,964 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:38,964 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:38,965 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A211CF8A9E55474097741C67742D40FF:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [562ED65A1AB745E2A886331788491D78:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:38,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:38,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A211CF8A9E55474097741C67742D40FF being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:38,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:38,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:38,965 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A211CF8A9E55474097741C67742D40FF: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:38,966 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A211CF8A9E55474097741C67742D40FF: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:38,967 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A211CF8A9E55474097741C67742D40FF with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:38,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A211CF8A9E55474097741C67742D40FF
2016-05-18 00:57:38,970 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:38,971 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533058689_0001_1_01_000000_0
2016-05-18 00:57:38,974 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:38,974 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A211CF8A9E55474097741C67742D40FF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:38,975 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: C092818E59E749A2BAB612A7CD7DABA5
2016-05-18 00:57:38,975 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_-1"], id: 562ED65A1AB745E2A886331788491D78
2016-05-18 00:57:38,976 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_-1"]
2016-05-18 00:57:38,976 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] C092818E59E749A2BAB612A7CD7DABA5, for 1 inputs
2016-05-18 00:57:38,978 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533058689_0001_1_00_000000_0_10147} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533058978, TimeTaken=2, Rate=0.09 MB/s
2016-05-18 00:57:38,978 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A211CF8A9E55474097741C67742D40FF
2016-05-18 00:57:38,978 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:38,979 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A211CF8A9E55474097741C67742D40FF, wasInterrupted=false
2016-05-18 00:57:38,979 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - A211CF8A9E55474097741C67742D40FF: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:38,980 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:38,981 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:38,982 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:38,982 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001_wd/localmode-local-dir/output/attempt_1463533058689_0001_1_00_000000_0_10147/file.out, len=181, isLocal=true
2016-05-18 00:57:38,982 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:38,982 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:38,982 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:38,983 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:38,983 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A211CF8A9E55474097741C67742D40FF
2016-05-18 00:57:38,983 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A211CF8A9E55474097741C67742D40FF: Shuffle Runner thread complete
2016-05-18 00:57:38,983 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B0D6D82DBC514BC5B868F2860E8A6262, all 1 inputs ready in: 00:00:00.007
2016-05-18 00:57:38,984 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:38,984 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A211CF8A9E55474097741C67742D40FF: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:38,994 INFO output.MROutput (MROutput.java:close(526)) - 562ED65A1AB745E2A886331788491D78 closed
2016-05-18 00:57:38,994 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533058689_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:38,995 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533058689_0001_1_01_000000_0
2016-05-18 00:57:38,995 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B0D6D82DBC514BC5B868F2860E8A6262, mem on close (mb), free: 150, total: 364, max: 672
2016-05-18 00:57:38,997 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A211CF8A9E55474097741C67742D40FF
2016-05-18 00:57:38,997 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A211CF8A9E55474097741C67742D40FF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:38,997 INFO output.MROutput (MROutput.java:close(526)) - 562ED65A1AB745E2A886331788491D78 closed
2016-05-18 00:57:38,997 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:38 Completed running task attempt: attempt_1463533058689_0001_1_01_000000_0
2016-05-18 00:57:39,000 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533058689_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:39,000 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533058689_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:39,000 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533058689_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533058983, Process_Duration=12, Process_End_Time=1463533058995, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:39,001 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533058689_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:39,001 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:39,001 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:39,001 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:39,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B0D6D82DBC514BC5B868F2860E8A6262, taskAttemptId=attempt_1463533058689_0001_1_01_000000_0, creationTime=1463533058932, allocationTime=1463533058933, startTime=1463533058935, finishTime=1463533059001, timeTaken=66, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:39,002 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533058689_0001_1_01_000000_0
2016-05-18 00:57:39,002 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:TASK_FINISHED]: vertexName=B0D6D82DBC514BC5B868F2860E8A6262, taskId=task_1463533058689_0001_1_01_000000, startTime=1463533058935, finishTime=1463533059002, timeTaken=67, status=SUCCEEDED, successfulAttemptID=attempt_1463533058689_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533058983, Process_Duration=12, Process_End_Time=1463533058995, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:39,003 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:39,003 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262]
2016-05-18 00:57:39,003 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:VERTEX_FINISHED]: vertexName=B0D6D82DBC514BC5B868F2860E8A6262, vertexId=vertex_1463533058689_0001_1_01, initRequestedTime=1463533058852, initedTime=1463533058856, startRequestedTime=1463533058864, startedTime=1463533058865, finishTime=1463533059003, timeTaken=138, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533058983, Process_Duration=12, Process_End_Time=1463533058995, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533058935, firstTasksToStart=[ task_1463533058689_0001_1_01_000000 ], lastTaskFinishTime=1463533059001, lastTasksToFinish=[ task_1463533058689_0001_1_01_000000 ], minTaskDuration=66, maxTaskDuration=66, avgTaskDuration=66.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533058689_0001_1_01_000000 ], longestDurationTasks=[ task_1463533058689_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:39,004 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:39,004 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533058689_0001_00_000002, stoppedTime=1463533059004, exitStatus=0
2016-05-18 00:57:39,004 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:39,004 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:39,004 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533058689_0001_1_00 [A211CF8A9E55474097741C67742D40FF]
2016-05-18 00:57:39,004 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533058689_0001_1
2016-05-18 00:57:39,004 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533058689_0001_1
2016-05-18 00:57:39,005 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 562ED65A1AB745E2A886331788491D78 for vertex: vertex_1463533058689_0001_1_01 [B0D6D82DBC514BC5B868F2860E8A6262], outputName: 562ED65A1AB745E2A886331788491D78
2016-05-18 00:57:39,005 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533058689_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:39,006 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533058689_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:39,008 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533058689_0001_00_000002
2016-05-18 00:57:39,008 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533058689_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:39,008 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533058689_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:39,008 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533058689_0001_00_000002, Exiting
2016-05-18 00:57:39,008 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533058689_0001_00_000002
2016-05-18 00:57:39,009 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533058689_0001_00_000002
2016-05-18 00:57:39,009 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533058689_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:39,009 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533058689_0001_00_000002 completed successfully
2016-05-18 00:57:39,013 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:562ED65A1AB745E2A886331788491D78 of vertex/vertexGroup:B0D6D82DBC514BC5B868F2860E8A6262 isVertexGroupOutput:false
2016-05-18 00:57:39,013 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:39,014 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533058689_0001_1, queueSize=0
2016-05-18 00:57:39,015 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533058689_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533058689_0001_1, startTime=1463533058849, finishTime=1463533059013, timeTaken=164, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=410, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066117898, Process_Duration=15, Process_End_Time=2927066117913, Read_Duration=4, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:39,015 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533058689_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:39,015 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533058689_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:39 Completed Dag: dag_1463533058689_0001_1
2016-05-18 00:57:39,015 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533058689_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:39,016 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533058689_0001_1_post
2016-05-18 00:57:39,016 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:39,016 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:39,016 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:39,016 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E] every/(1/1) ...mblyplatform/0_UNKNOWN_-1, with id=dag_1463533058689_0001_1
2016-05-18 00:57:39,016 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E] every/(1/1) ...mblyplatform/0_UNKNOWN_-1, with id=dag_1463533058689_0001_1
2016-05-18 00:57:39,022 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...mblyplatform/0_UNKNOWN_-1, applicationId=application_1463533058689_0001
2016-05-18 00:57:39,024 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:39,024 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:39,024 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:39,024 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:39,024 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:39,025 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:39,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:39,025 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:39,025 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:39,025 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:39,026 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:39,026 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:39,026 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:39,026 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:39,029 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/0AE2C7A593D74C29A7B549F624E136EC/F6A5CEB75861473C9F96314CFADE4E4E/.tez/application_1463533058689_0001
2016-05-18 00:57:39,030 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_-1/_temporary
2016-05-18 00:57:39,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.359
2016-05-18 00:57:39,109 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_UNKNOWN_-1
2016-05-18 00:57:39,131 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:39,131 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:39,160 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.003
2016-05-18 00:57:39,161 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.004
2016-05-18 00:57:39,161 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:39,162 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:39,162 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_UNKNOWN_'field'
2016-05-18 00:57:39,185 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:39,185 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:39,219 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:39,219 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:39,242 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.008
2016-05-18 00:57:39,257 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.023
2016-05-18 00:57:39,257 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:39,257 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:39,258 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:39,274 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:39,280 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_RESULTS
2016-05-18 00:57:39,325 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:39,325 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:39,325 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_RESULTS"]
2016-05-18 00:57:39,325 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:39,325 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:39,325 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:39,327 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...latform/0_UNKNOWN_RESULTS
2016-05-18 00:57:39,328 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:39,341 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:39,346 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:39,346 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:39,346 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:39,357 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001 doesn't exist and is created
2016-05-18 00:57:39,372 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:39,372 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:39,373 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001_wd
2016-05-18 00:57:39,383 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533059346_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:39,383 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:39,383 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:39,384 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:39,384 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:39,385 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:39,385 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:39,385 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:39,387 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:39,387 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:39,387 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:39,387 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:39,387 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:39,387 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:39,388 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:39,388 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:39,388 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:39,388 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:39,388 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:39,388 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:39,390 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:39,391 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:39,391 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:39,391 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001_wd/localmode-log-dir
2016-05-18 00:57:39,392 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001_wd/localmode-log-dir/history.txt.appattempt_1463533059346_0001_000000, maxErrors=10
2016-05-18 00:57:39,392 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:39,392 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533059346_0001_000000, appSubmitTime=1463533059382, launchTime=1463533059382
2016-05-18 00:57:39,394 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:39,399 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:54396
2016-05-18 00:57:39,399 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533059346
2016-05-18 00:57:39,403 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533059346_0001_000000, startTime=1463533059402
2016-05-18 00:57:39,403 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:39,473 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:39,473 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:39,473 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...latform/0_UNKNOWN_RESULTS, applicationId=application_1463533059346_0001, dagName=[B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9] every/(1/1) ...latform/0_UNKNOWN_RESULTS
2016-05-18 00:57:39,490 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9] every/(1/1) ...latform/0_UNKNOWN_RESULTS
2016-05-18 00:57:39,490 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:39,490 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533059346_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001_wd/localmode-log-dir/dag_1463533059346_0001_1.dot
2016-05-18 00:57:39,491 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001_wd/localmode-log-dir/dag_1463533059346_0001_1-tez-dag.pb.txt
2016-05-18 00:57:39,497 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533059346_0001_1
2016-05-18 00:57:39,497 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:39,497 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9] every/(1/1) ...latform/0_UNKNOWN_RESULTS
2016-05-18 00:57:39 Running Dag: dag_1463533059346_0001_1
2016-05-18 00:57:39,506 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533059346_0001_1, submitTime=1463533059490
2016-05-18 00:57:39,507 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001/tez-conf.pb" } size: 36913 timestamp: 1463533059000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:39,507 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,507 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:39,507 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001/tez-conf.pb" } size: 36913 timestamp: 1463533059000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:39,507 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]
2016-05-18 00:57:39,508 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:39,508 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:39,508 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533059346_0001_1, initTime=1463533059508
2016-05-18 00:57:39,508 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533059346_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:39,509 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:39,509 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:DAG_STARTED]: dagID=dag_1463533059346_0001_1, startTime=1463533059509
2016-05-18 00:57:39,509 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533059346_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:39,509 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D02282608DF843BABF9F29BE283604F1 : {CC4ABBA659D04B539637182D92BAF1FF={InputName=CC4ABBA659D04B539637182D92BAF1FF}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:39,509 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: CC4ABBA659D04B539637182D92BAF1FF, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:39,509 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,509 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,509 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,509 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,510 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:39,512 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]
2016-05-18 00:57:39,512 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...latform/0_UNKNOWN_RESULTS, applicationId=application_1463533059346_0001, dagName=[B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9] every/(1/1) ...latform/0_UNKNOWN_RESULTS
2016-05-18 00:57:39,513 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: CC4ABBA659D04B539637182D92BAF1FF on vertex vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,515 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:39,515 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]
2016-05-18 00:57:39,515 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input CC4ABBA659D04B539637182D92BAF1FF asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:39,515 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]
2016-05-18 00:57:39,516 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533059346_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:39,516 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253], numAdditionalOutputs=1
2016-05-18 00:57:39,516 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:39,516 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:39,516 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=6F3F8C51D47E4D7D903725E0BD73AA04, vertex=vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:39,517 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:39,517 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:39,517 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: CC4ABBA659D04B539637182D92BAF1FF on vertex vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,518 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for FE24F7433B994360AEAA211BF0B7B253:6F3F8C51D47E4D7D903725E0BD73AA04 using oldmapred API
2016-05-18 00:57:39,518 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=6F3F8C51D47E4D7D903725E0BD73AA04, vertexName=FE24F7433B994360AEAA211BF0B7B253, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:39,519 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FE24F7433B994360AEAA211BF0B7B253, vertexId=vertex_1463533059346_0001_1_01, initRequestedTime=1463533059512, initedTime=1463533059516, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:39,519 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:39,521 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:39,521 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {CC4ABBA659D04B539637182D92BAF1FF=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:39,523 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] parallelism set to 1
2016-05-18 00:57:39,524 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:39,524 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D02282608DF843BABF9F29BE283604F1, vertexId=vertex_1463533059346_0001_1_00, initRequestedTime=1463533059509, initedTime=1463533059524, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:39,524 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] with distanceFromRoot: 0
2016-05-18 00:57:39,524 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:39,525 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533059346_0001_1_00, startRequestedTime=1463533059511, startedTime=1463533059525
2016-05-18 00:57:39,525 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533059346_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:39,526 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:39,526 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533059346_0001_1_00 for vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253] numStartedSources: 1 numSources: 1
2016-05-18 00:57:39,526 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D02282608DF843BABF9F29BE283604F1
2016-05-18 00:57:39,526 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,526 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253] with distanceFromRoot: 1
2016-05-18 00:57:39,527 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533059346_0001_1_01, startRequestedTime=1463533059526, startedTime=1463533059526
2016-05-18 00:57:39,527 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:39,530 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:TASK_STARTED]: vertexName=D02282608DF843BABF9F29BE283604F1, taskId=task_1463533059346_0001_1_00_000000, scheduledTime=1463533059527, launchTime=1463533059527
2016-05-18 00:57:39,531 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,531 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: FE24F7433B994360AEAA211BF0B7B253 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:39,531 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D02282608DF843BABF9F29BE283604F1 in vertex: FE24F7433B994360AEAA211BF0B7B253 numBipartiteSourceTasks: 1
2016-05-18 00:57:39,532 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:39,533 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533059346_0001_00_000001, launchTime=1463533059533
2016-05-18 00:57:39,534 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533059346_0001_00_000001
2016-05-18 00:57:39,535 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533059346_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533059346_0001_1_00_000000_0
2016-05-18 00:57:39,535 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533059346_0001_1_00_000000_0] started. Is using containerId: [container_1463533059346_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:39,535 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D02282608DF843BABF9F29BE283604F1, taskAttemptId=attempt_1463533059346_0001_1_00_000000_0, startTime=1463533059535, containerId=container_1463533059346_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:39 Starting to run new task attempt: attempt_1463533059346_0001_1_00_000000_0
2016-05-18 00:57:39,537 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533059346_0001_1_00_000000_0
2016-05-18 00:57:39,537 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533059346_0001, with dag id: dag_1463533059346_0001_1
2016-05-18 00:57:39,537 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:39,538 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:39,538 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9] every/(1/1) ...latform/0_UNKNOWN_RESULTS, VertexName: D02282608DF843BABF9F29BE283604F1, VertexParallelism: 1, TaskAttemptID:attempt_1463533059346_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CC4ABBA659D04B539637182D92BAF1FF, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FE24F7433B994360AEAA211BF0B7B253, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:39,539 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:39,540 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533059346_0001_1_00_000000_0
2016-05-18 00:57:39,543 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:39,543 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533059346_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:39,543 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533059346_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:39,544 INFO input.MRInput (MRInput.java:initialize(440)) - CC4ABBA659D04B539637182D92BAF1FF using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:39,544 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: CC4ABBA659D04B539637182D92BAF1FF
2016-05-18 00:57:39,548 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D02282608DF843BABF9F29BE283604F1, ordinal: 0
2016-05-18 00:57:39,548 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D02282608DF843BABF9F29BE283604F1, mem on start (mb), free: 231, total: 365, max: 672
2016-05-18 00:57:39,548 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:39,548 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:39,548 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:39,548 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:39,548 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[CC4ABBA659D04B539637182D92BAF1FF:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FE24F7433B994360AEAA211BF0B7B253:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:39,549 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:39,549 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:39,549 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:39,549 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[CC4ABBA659D04B539637182D92BAF1FF:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FE24F7433B994360AEAA211BF0B7B253:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:39,549 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:39,549 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CC4ABBA659D04B539637182D92BAF1FF being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:39,549 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:39,549 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:39,549 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CC4ABBA659D04B539637182D92BAF1FF
2016-05-18 00:57:39,550 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:39,550 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533059346_0001_1_00_000000_0
2016-05-18 00:57:39,553 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:39,554 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:39,554 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: CC4ABBA659D04B539637182D92BAF1FF
2016-05-18 00:57:39,554 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: EF5ECCA354804D72B7C3BDC748A029EE
2016-05-18 00:57:39,555 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] EF5ECCA354804D72B7C3BDC748A029EE
2016-05-18 00:57:39,556 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:39,554 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - CC4ABBA659D04B539637182D92BAF1FF initialized RecordReader from event
2016-05-18 00:57:39,560 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:39,561 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FE24F7433B994360AEAA211BF0B7B253 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@58b230b5, comparator=cascading.tuple.hadoop.util.TupleComparator@1a9be891, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:39,561 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FE24F7433B994360AEAA211BF0B7B253: , UsingHashComparator=false
2016-05-18 00:57:39,574 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:39,574 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:39,574 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FE24F7433B994360AEAA211BF0B7B253: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:39,575 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FE24F7433B994360AEAA211BF0B7B253: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:39,575 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:39,575 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D02282608DF843BABF9F29BE283604F1, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:39,577 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533059346_0001_1_00_000000_0
2016-05-18 00:57:39,578 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D02282608DF843BABF9F29BE283604F1, mem on close (mb), free: 129, total: 365, max: 672
2016-05-18 00:57:39,578 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FE24F7433B994360AEAA211BF0B7B253: Starting flush of map output
2016-05-18 00:57:39,578 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FE24F7433B994360AEAA211BF0B7B253: Span0.length = 20, perItem = 6
2016-05-18 00:57:39,578 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FE24F7433B994360AEAA211BF0B7B253: done sorting span=0, length=20, time=0
2016-05-18 00:57:39,578 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FE24F7433B994360AEAA211BF0B7B253: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:39,579 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FE24F7433B994360AEAA211BF0B7B253: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001_wd/localmode-local-dir/output/attempt_1463533059346_0001_1_00_000000_0_10152_0/file.out
2016-05-18 00:57:39,583 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:39 Completed running task attempt: attempt_1463533059346_0001_1_00_000000_0
2016-05-18 00:57:39,583 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533059346_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:39,584 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533059346_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:39,584 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533059346_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533059575, Process_Duration=2, Process_End_Time=1463533059577, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:39,584 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533059346_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:39,584 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:39,585 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D02282608DF843BABF9F29BE283604F1, taskAttemptId=attempt_1463533059346_0001_1_00_000000_0, creationTime=1463533059531, allocationTime=1463533059533, startTime=1463533059535, finishTime=1463533059584, timeTaken=49, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:39,585 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:39,585 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:39,585 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533059346_0001_1_00_000000_0
2016-05-18 00:57:39,586 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:TASK_FINISHED]: vertexName=D02282608DF843BABF9F29BE283604F1, taskId=task_1463533059346_0001_1_00_000000, startTime=1463533059535, finishTime=1463533059586, timeTaken=51, status=SUCCEEDED, successfulAttemptID=attempt_1463533059346_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533059575, Process_Duration=2, Process_End_Time=1463533059577, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:39,586 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:39,587 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,587 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:VERTEX_FINISHED]: vertexName=D02282608DF843BABF9F29BE283604F1, vertexId=vertex_1463533059346_0001_1_00, initRequestedTime=1463533059509, initedTime=1463533059524, startRequestedTime=1463533059511, startedTime=1463533059525, finishTime=1463533059587, timeTaken=62, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533059575, Process_Duration=2, Process_End_Time=1463533059577, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533059535, firstTasksToStart=[ task_1463533059346_0001_1_00_000000 ], lastTaskFinishTime=1463533059584, lastTasksToFinish=[ task_1463533059346_0001_1_00_000000 ], minTaskDuration=49, maxTaskDuration=49, avgTaskDuration=49.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533059346_0001_1_00_000000 ], longestDurationTasks=[ task_1463533059346_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:39,587 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:39,587 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533059346_0001_00_000001, stoppedTime=1463533059587, exitStatus=0
2016-05-18 00:57:39,587 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253] attempt: attempt_1463533059346_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:39,587 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:39,588 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:39,588 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FE24F7433B994360AEAA211BF0B7B253
2016-05-18 00:57:39,588 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533059346_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:39,588 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D02282608DF843BABF9F29BE283604F1, destinationVertexName=FE24F7433B994360AEAA211BF0B7B253. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:39,588 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]
2016-05-18 00:57:39,588 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:TASK_STARTED]: vertexName=FE24F7433B994360AEAA211BF0B7B253, taskId=task_1463533059346_0001_1_01_000000, scheduledTime=1463533059588, launchTime=1463533059588
2016-05-18 00:57:39,588 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533059346_0001_00_000001
2016-05-18 00:57:39,588 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]
2016-05-18 00:57:39,589 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533059346_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:39,589 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533059346_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:39,590 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533059346_0001_00_000001, Exiting
2016-05-18 00:57:39,590 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533059346_0001_00_000001
2016-05-18 00:57:39,590 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533059346_0001_00_000001
2016-05-18 00:57:39,590 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533059346_0001_00_000002, launchTime=1463533059590
2016-05-18 00:57:39,591 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533059346_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:39,591 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533059346_0001_00_000002
2016-05-18 00:57:39,592 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533059346_0001_00_000001 completed successfully
2016-05-18 00:57:39,592 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533059346_0001_1_01_000000_0] started. Is using containerId: [container_1463533059346_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:39,592 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FE24F7433B994360AEAA211BF0B7B253, taskAttemptId=attempt_1463533059346_0001_1_01_000000_0, startTime=1463533059592, containerId=container_1463533059346_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:39,593 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533059346_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533059346_0001_1_01_000000_0
2016-05-18 00:57:39 Starting to run new task attempt: attempt_1463533059346_0001_1_01_000000_0
2016-05-18 00:57:39,593 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533059346_0001_1_01_000000_0
2016-05-18 00:57:39,594 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:39,594 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:39,594 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9] every/(1/1) ...latform/0_UNKNOWN_RESULTS, VertexName: FE24F7433B994360AEAA211BF0B7B253, VertexParallelism: 1, TaskAttemptID:attempt_1463533059346_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D02282608DF843BABF9F29BE283604F1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=6F3F8C51D47E4D7D903725E0BD73AA04, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:39,595 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:39,595 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533059346_0001_1_01_000000_0
2016-05-18 00:57:39,595 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533059346_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]
2016-05-18 00:57:39,596 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533059346_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:39,598 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:39,598 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:39,603 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FE24F7433B994360AEAA211BF0B7B253, ordinal: 1
2016-05-18 00:57:39,603 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FE24F7433B994360AEAA211BF0B7B253, mem on start (mb), free: 124, total: 365, max: 672
2016-05-18 00:57:39,603 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:39,603 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:39,603 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:39,606 INFO output.MROutput (MROutput.java:initialize(421)) - 6F3F8C51D47E4D7D903725E0BD73AA04: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:39,607 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:39,607 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D02282608DF843BABF9F29BE283604F1:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [6F3F8C51D47E4D7D903725E0BD73AA04:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:39,607 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:39,607 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:39,607 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:39,608 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D02282608DF843BABF9F29BE283604F1:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [6F3F8C51D47E4D7D903725E0BD73AA04:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:39,608 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:39,608 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D02282608DF843BABF9F29BE283604F1 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:39,608 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:39,609 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:39,609 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D02282608DF843BABF9F29BE283604F1: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:39,609 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D02282608DF843BABF9F29BE283604F1: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:39,610 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D02282608DF843BABF9F29BE283604F1 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:39,613 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D02282608DF843BABF9F29BE283604F1
2016-05-18 00:57:39,613 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:39,613 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533059346_0001_1_01_000000_0
2016-05-18 00:57:39,613 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D02282608DF843BABF9F29BE283604F1: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:39,616 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:39,616 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: EF5ECCA354804D72B7C3BDC748A029EE
2016-05-18 00:57:39,616 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_RESULTS"], id: 6F3F8C51D47E4D7D903725E0BD73AA04
2016-05-18 00:57:39,617 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_RESULTS"]
2016-05-18 00:57:39,617 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533059346_0001_1_00_000000_0_10152} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533059617, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:39,617 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] EF5ECCA354804D72B7C3BDC748A029EE, for 1 inputs
2016-05-18 00:57:39,617 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D02282608DF843BABF9F29BE283604F1
2016-05-18 00:57:39,618 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:39,618 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D02282608DF843BABF9F29BE283604F1, wasInterrupted=false
2016-05-18 00:57:39,618 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:39,619 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:39,619 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:39,620 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001_wd/localmode-local-dir/output/attempt_1463533059346_0001_1_00_000000_0_10152/file.out, len=181, isLocal=true
2016-05-18 00:57:39,620 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:39,620 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:39,620 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:39,620 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:39,620 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D02282608DF843BABF9F29BE283604F1
2016-05-18 00:57:39,620 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D02282608DF843BABF9F29BE283604F1: Shuffle Runner thread complete
2016-05-18 00:57:39,621 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FE24F7433B994360AEAA211BF0B7B253, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:39,622 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:39,622 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D02282608DF843BABF9F29BE283604F1: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:39,626 INFO output.MROutput (MROutput.java:close(526)) - 6F3F8C51D47E4D7D903725E0BD73AA04 closed
2016-05-18 00:57:39,626 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533059346_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:39,627 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533059346_0001_1_01_000000_0
2016-05-18 00:57:39,627 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FE24F7433B994360AEAA211BF0B7B253, mem on close (mb), free: 119, total: 365, max: 672
2016-05-18 00:57:39,627 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D02282608DF843BABF9F29BE283604F1
2016-05-18 00:57:39,627 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D02282608DF843BABF9F29BE283604F1: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:39,628 INFO output.MROutput (MROutput.java:close(526)) - 6F3F8C51D47E4D7D903725E0BD73AA04 closed
2016-05-18 00:57:39,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:39 Completed running task attempt: attempt_1463533059346_0001_1_01_000000_0
2016-05-18 00:57:39,628 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533059346_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:39,628 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533059346_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:39,629 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533059346_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533059621, Process_Duration=6, Process_End_Time=1463533059627, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:39,629 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533059346_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:39,629 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:39,630 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:39,630 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:39,630 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FE24F7433B994360AEAA211BF0B7B253, taskAttemptId=attempt_1463533059346_0001_1_01_000000_0, creationTime=1463533059588, allocationTime=1463533059589, startTime=1463533059592, finishTime=1463533059630, timeTaken=38, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:39,630 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533059346_0001_1_01_000000_0
2016-05-18 00:57:39,630 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533059346_0001_00_000002
2016-05-18 00:57:39,630 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:TASK_FINISHED]: vertexName=FE24F7433B994360AEAA211BF0B7B253, taskId=task_1463533059346_0001_1_01_000000, startTime=1463533059592, finishTime=1463533059630, timeTaken=38, status=SUCCEEDED, successfulAttemptID=attempt_1463533059346_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533059621, Process_Duration=6, Process_End_Time=1463533059627, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:39,632 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:39,632 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253]
2016-05-18 00:57:39,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:VERTEX_FINISHED]: vertexName=FE24F7433B994360AEAA211BF0B7B253, vertexId=vertex_1463533059346_0001_1_01, initRequestedTime=1463533059512, initedTime=1463533059516, startRequestedTime=1463533059526, startedTime=1463533059526, finishTime=1463533059632, timeTaken=106, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533059621, Process_Duration=6, Process_End_Time=1463533059627, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533059592, firstTasksToStart=[ task_1463533059346_0001_1_01_000000 ], lastTaskFinishTime=1463533059630, lastTasksToFinish=[ task_1463533059346_0001_1_01_000000 ], minTaskDuration=38, maxTaskDuration=38, avgTaskDuration=38.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533059346_0001_1_01_000000 ], longestDurationTasks=[ task_1463533059346_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:39,632 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:39,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533059346_0001_00_000002, stoppedTime=1463533059632, exitStatus=0
2016-05-18 00:57:39,633 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:39,633 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:39,633 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533059346_0001_1_00 [D02282608DF843BABF9F29BE283604F1]
2016-05-18 00:57:39,633 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533059346_0001_1
2016-05-18 00:57:39,633 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533059346_0001_1
2016-05-18 00:57:39,633 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533059346_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:39,633 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533059346_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:39,633 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 6F3F8C51D47E4D7D903725E0BD73AA04 for vertex: vertex_1463533059346_0001_1_01 [FE24F7433B994360AEAA211BF0B7B253], outputName: 6F3F8C51D47E4D7D903725E0BD73AA04
2016-05-18 00:57:39,639 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:6F3F8C51D47E4D7D903725E0BD73AA04 of vertex/vertexGroup:FE24F7433B994360AEAA211BF0B7B253 isVertexGroupOutput:false
2016-05-18 00:57:39,639 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:39,641 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533059346_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:39,641 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533059346_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:39,641 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533059346_0001_00_000002, Exiting
2016-05-18 00:57:39,641 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533059346_0001_00_000002
2016-05-18 00:57:39,641 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533059346_0001_00_000002
2016-05-18 00:57:39,641 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533059346_0001_1, queueSize=0
2016-05-18 00:57:39,641 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533059346_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:39,641 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533059346_0001_00_000002 completed successfully
2016-05-18 00:57:39,642 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059346_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533059346_0001_1, startTime=1463533059509, finishTime=1463533059640, timeTaken=131, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=230, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066119196, Process_Duration=8, Process_End_Time=2927066119204, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:39,642 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533059346_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:39,642 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533059346_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:39 Completed Dag: dag_1463533059346_0001_1
2016-05-18 00:57:39,642 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533059346_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:39,642 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533059346_0001_1_post
2016-05-18 00:57:39,642 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:39,643 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:39,643 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:39,643 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9] every/(1/1) ...latform/0_UNKNOWN_RESULTS, with id=dag_1463533059346_0001_1
2016-05-18 00:57:39,643 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9] every/(1/1) ...latform/0_UNKNOWN_RESULTS, with id=dag_1463533059346_0001_1
2016-05-18 00:57:39,651 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...latform/0_UNKNOWN_RESULTS, applicationId=application_1463533059346_0001
2016-05-18 00:57:39,652 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:39,652 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:39,652 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:39,652 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:39,653 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:39,653 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:39,653 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:39,654 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:39,654 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:39,654 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:39,654 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:39,654 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:39,654 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:39,655 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:39,659 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B454D44A7B7841218AFE819B1080E97E/E491F4BF7BDF463DBD9EC91A068A55D9/.tez/application_1463533059346_0001
2016-05-18 00:57:39,659 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_RESULTS/_temporary
2016-05-18 00:57:39,659 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.334
2016-05-18 00:57:39,735 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_UNKNOWN_RESULTS
2016-05-18 00:57:39,758 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:39,758 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:39,791 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 00:57:39,798 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 00:57:39,798 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:39,798 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:39,799 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:39,813 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:39,818 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_ALL
2016-05-18 00:57:39,864 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:39,864 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:39,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_ALL"]
2016-05-18 00:57:39,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:39,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:39,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:39,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...blyplatform/0_UNKNOWN_ALL
2016-05-18 00:57:39,866 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:39,879 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:39,883 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:39,883 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:39,883 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:39,885 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:39,885 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:39,892 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001 doesn't exist and is created
2016-05-18 00:57:39,906 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:39,907 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:39,907 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001_wd
2016-05-18 00:57:39,917 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533059883_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:39,917 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:39,917 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:39,918 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:39,918 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:39,918 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:39,919 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:39,919 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:39,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:39,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:39,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:39,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:39,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:39,921 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:39,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:39,922 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:39,922 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:39,922 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:39,922 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:39,922 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:39,924 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:39,925 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:39,925 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:39,925 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001_wd/localmode-log-dir
2016-05-18 00:57:39,926 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001_wd/localmode-log-dir/history.txt.appattempt_1463533059883_0001_000000, maxErrors=10
2016-05-18 00:57:39,926 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:39,926 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533059883_0001_000000, appSubmitTime=1463533059916, launchTime=1463533059916
2016-05-18 00:57:39,928 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:39,933 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:52371
2016-05-18 00:57:39,933 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533059883
2016-05-18 00:57:39,935 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533059883_0001_000000, startTime=1463533059935
2016-05-18 00:57:39,935 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:40,007 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:40,007 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:40,008 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...blyplatform/0_UNKNOWN_ALL, applicationId=application_1463533059883_0001, dagName=[757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826] every/(1/1) ...blyplatform/0_UNKNOWN_ALL
2016-05-18 00:57:40,023 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826] every/(1/1) ...blyplatform/0_UNKNOWN_ALL
2016-05-18 00:57:40,023 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:40,024 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533059883_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001_wd/localmode-log-dir/dag_1463533059883_0001_1.dot
2016-05-18 00:57:40,024 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001_wd/localmode-log-dir/dag_1463533059883_0001_1-tez-dag.pb.txt
2016-05-18 00:57:40,032 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533059883_0001_1
2016-05-18 00:57:40,032 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,032 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826] every/(1/1) ...blyplatform/0_UNKNOWN_ALL
2016-05-18 00:57:40 Running Dag: dag_1463533059883_0001_1
2016-05-18 00:57:40,045 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533059883_0001_1, submitTime=1463533060023
2016-05-18 00:57:40,046 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001/tez-conf.pb" } size: 36913 timestamp: 1463533059000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:40,046 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,047 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:40,047 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001/tez-conf.pb" } size: 36913 timestamp: 1463533059000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:40,047 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]
2016-05-18 00:57:40,048 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:40,048 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:40,048 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533059883_0001_1, initTime=1463533060048
2016-05-18 00:57:40,049 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533059883_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:40,049 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:40,050 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:DAG_STARTED]: dagID=dag_1463533059883_0001_1, startTime=1463533060050
2016-05-18 00:57:40,050 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533059883_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:40,050 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: DF574E3E74A1448C814CE3F445904655 : {232C3CCDBA694ABABFC0D9054750CC67={InputName=232C3CCDBA694ABABFC0D9054750CC67}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:40,050 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 232C3CCDBA694ABABFC0D9054750CC67, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:40,050 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,050 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...blyplatform/0_UNKNOWN_ALL, applicationId=application_1463533059883_0001, dagName=[757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826] every/(1/1) ...blyplatform/0_UNKNOWN_ALL
2016-05-18 00:57:40,050 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,050 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,051 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,054 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:40,054 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]
2016-05-18 00:57:40,057 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 232C3CCDBA694ABABFC0D9054750CC67 on vertex vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,057 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:40,058 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]
2016-05-18 00:57:40,058 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]
2016-05-18 00:57:40,058 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533059883_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:40,058 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5], numAdditionalOutputs=1
2016-05-18 00:57:40,058 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=D7654B6BD38646A7B52BD395671868C1, vertex=vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:40,059 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 232C3CCDBA694ABABFC0D9054750CC67 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:40,060 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:40,060 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:40,060 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:40,060 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:40,060 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for FC831479FD67477EB1D51EFC28E077F5:D7654B6BD38646A7B52BD395671868C1 using oldmapred API
2016-05-18 00:57:40,061 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 232C3CCDBA694ABABFC0D9054750CC67 on vertex vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,061 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=D7654B6BD38646A7B52BD395671868C1, vertexName=FC831479FD67477EB1D51EFC28E077F5, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:40,062 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FC831479FD67477EB1D51EFC28E077F5, vertexId=vertex_1463533059883_0001_1_01, initRequestedTime=1463533060054, initedTime=1463533060058, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:40,062 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:40,066 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:40,066 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {232C3CCDBA694ABABFC0D9054750CC67=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:40,067 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] parallelism set to 1
2016-05-18 00:57:40,067 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:40,067 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DF574E3E74A1448C814CE3F445904655, vertexId=vertex_1463533059883_0001_1_00, initRequestedTime=1463533060050, initedTime=1463533060067, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:40,067 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] with distanceFromRoot: 0
2016-05-18 00:57:40,068 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:40,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533059883_0001_1_00, startRequestedTime=1463533060054, startedTime=1463533060068
2016-05-18 00:57:40,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533059883_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:40,068 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:40,068 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533059883_0001_1_00 for vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5] numStartedSources: 1 numSources: 1
2016-05-18 00:57:40,068 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in DF574E3E74A1448C814CE3F445904655
2016-05-18 00:57:40,068 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5] with distanceFromRoot: 1
2016-05-18 00:57:40,070 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,071 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533059883_0001_1_01, startRequestedTime=1463533060068, startedTime=1463533060070
2016-05-18 00:57:40,071 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:40,071 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:TASK_STARTED]: vertexName=DF574E3E74A1448C814CE3F445904655, taskId=task_1463533059883_0001_1_00_000000, scheduledTime=1463533060071, launchTime=1463533060071
2016-05-18 00:57:40,072 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,073 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: FC831479FD67477EB1D51EFC28E077F5 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:40,074 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: DF574E3E74A1448C814CE3F445904655 in vertex: FC831479FD67477EB1D51EFC28E077F5 numBipartiteSourceTasks: 1
2016-05-18 00:57:40,074 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:40,076 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533059883_0001_00_000001, launchTime=1463533060076
2016-05-18 00:57:40,076 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533059883_0001_00_000001
2016-05-18 00:57:40,077 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533059883_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533059883_0001_1_00_000000_0
2016-05-18 00:57:40,077 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533059883_0001_1_00_000000_0] started. Is using containerId: [container_1463533059883_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:40,077 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DF574E3E74A1448C814CE3F445904655, taskAttemptId=attempt_1463533059883_0001_1_00_000000_0, startTime=1463533060077, containerId=container_1463533059883_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:40 Starting to run new task attempt: attempt_1463533059883_0001_1_00_000000_0
2016-05-18 00:57:40,078 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533059883_0001, with dag id: dag_1463533059883_0001_1
2016-05-18 00:57:40,078 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533059883_0001_1_00_000000_0
2016-05-18 00:57:40,078 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,079 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:40,079 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826] every/(1/1) ...blyplatform/0_UNKNOWN_ALL, VertexName: DF574E3E74A1448C814CE3F445904655, VertexParallelism: 1, TaskAttemptID:attempt_1463533059883_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=232C3CCDBA694ABABFC0D9054750CC67, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FC831479FD67477EB1D51EFC28E077F5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:40,080 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:40,082 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533059883_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:40,083 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533059883_0001_1_00_000000_0
2016-05-18 00:57:40,083 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533059883_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:40,087 INFO input.MRInput (MRInput.java:initialize(440)) - 232C3CCDBA694ABABFC0D9054750CC67 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:40,087 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 232C3CCDBA694ABABFC0D9054750CC67
2016-05-18 00:57:40,088 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:40,091 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DF574E3E74A1448C814CE3F445904655, ordinal: 0
2016-05-18 00:57:40,092 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DF574E3E74A1448C814CE3F445904655, mem on start (mb), free: 214, total: 365, max: 672
2016-05-18 00:57:40,092 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:40,092 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:40,093 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:40,093 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:40,093 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[232C3CCDBA694ABABFC0D9054750CC67:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FC831479FD67477EB1D51EFC28E077F5:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:40,093 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:40,093 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:40,094 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:40,094 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[232C3CCDBA694ABABFC0D9054750CC67:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FC831479FD67477EB1D51EFC28E077F5:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:40,094 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:40,094 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 232C3CCDBA694ABABFC0D9054750CC67 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:40,094 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:40,094 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:40,095 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 232C3CCDBA694ABABFC0D9054750CC67
2016-05-18 00:57:40,095 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:40,096 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533059883_0001_1_00_000000_0
2016-05-18 00:57:40,097 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:40,098 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 232C3CCDBA694ABABFC0D9054750CC67 initialized RecordReader from event
2016-05-18 00:57:40,098 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:40,100 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 232C3CCDBA694ABABFC0D9054750CC67
2016-05-18 00:57:40,100 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: E75539C6ABC34BBFA1C07A09130A034A
2016-05-18 00:57:40,100 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] E75539C6ABC34BBFA1C07A09130A034A
2016-05-18 00:57:40,102 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:40,104 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:40,104 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FC831479FD67477EB1D51EFC28E077F5 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@640e702c, comparator=cascading.tuple.hadoop.util.TupleComparator@6fe68c05, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:40,105 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FC831479FD67477EB1D51EFC28E077F5: , UsingHashComparator=false
2016-05-18 00:57:40,117 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:40,118 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:40,118 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FC831479FD67477EB1D51EFC28E077F5: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:40,118 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FC831479FD67477EB1D51EFC28E077F5: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:40,119 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:40,120 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DF574E3E74A1448C814CE3F445904655, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:40,122 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533059883_0001_1_00_000000_0
2016-05-18 00:57:40,123 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DF574E3E74A1448C814CE3F445904655, mem on close (mb), free: 112, total: 365, max: 672
2016-05-18 00:57:40,123 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FC831479FD67477EB1D51EFC28E077F5: Starting flush of map output
2016-05-18 00:57:40,123 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FC831479FD67477EB1D51EFC28E077F5: Span0.length = 20, perItem = 6
2016-05-18 00:57:40,124 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FC831479FD67477EB1D51EFC28E077F5: done sorting span=0, length=20, time=0
2016-05-18 00:57:40,124 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FC831479FD67477EB1D51EFC28E077F5: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:40,125 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FC831479FD67477EB1D51EFC28E077F5: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001_wd/localmode-local-dir/output/attempt_1463533059883_0001_1_00_000000_0_10159_0/file.out
2016-05-18 00:57:40,129 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:40 Completed running task attempt: attempt_1463533059883_0001_1_00_000000_0
2016-05-18 00:57:40,130 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533059883_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:40,131 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533059883_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:40,131 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533059883_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533060120, Process_Duration=2, Process_End_Time=1463533060122, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:40,132 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533059883_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:40,132 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DF574E3E74A1448C814CE3F445904655, taskAttemptId=attempt_1463533059883_0001_1_00_000000_0, creationTime=1463533060071, allocationTime=1463533060075, startTime=1463533060077, finishTime=1463533060132, timeTaken=55, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:40,132 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533059883_0001_1_00_000000_0
2016-05-18 00:57:40,132 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:40,134 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:40,134 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,134 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533059883_0001_00_000001
2016-05-18 00:57:40,133 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:TASK_FINISHED]: vertexName=DF574E3E74A1448C814CE3F445904655, taskId=task_1463533059883_0001_1_00_000000, startTime=1463533060077, finishTime=1463533060133, timeTaken=56, status=SUCCEEDED, successfulAttemptID=attempt_1463533059883_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533060120, Process_Duration=2, Process_End_Time=1463533060122, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:40,135 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:40,135 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,135 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:VERTEX_FINISHED]: vertexName=DF574E3E74A1448C814CE3F445904655, vertexId=vertex_1463533059883_0001_1_00, initRequestedTime=1463533060050, initedTime=1463533060067, startRequestedTime=1463533060054, startedTime=1463533060068, finishTime=1463533060135, timeTaken=67, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533060120, Process_Duration=2, Process_End_Time=1463533060122, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533060077, firstTasksToStart=[ task_1463533059883_0001_1_00_000000 ], lastTaskFinishTime=1463533060132, lastTasksToFinish=[ task_1463533059883_0001_1_00_000000 ], minTaskDuration=55, maxTaskDuration=55, avgTaskDuration=55.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533059883_0001_1_00_000000 ], longestDurationTasks=[ task_1463533059883_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:40,135 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:40,136 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533059883_0001_00_000001, stoppedTime=1463533060136, exitStatus=0
2016-05-18 00:57:40,136 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5] attempt: attempt_1463533059883_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:40,136 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:40,136 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:40,136 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FC831479FD67477EB1D51EFC28E077F5
2016-05-18 00:57:40,136 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533059883_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:40,136 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=DF574E3E74A1448C814CE3F445904655, destinationVertexName=FC831479FD67477EB1D51EFC28E077F5. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:40,138 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]
2016-05-18 00:57:40,138 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:TASK_STARTED]: vertexName=FC831479FD67477EB1D51EFC28E077F5, taskId=task_1463533059883_0001_1_01_000000, scheduledTime=1463533060138, launchTime=1463533060138
2016-05-18 00:57:40,138 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]
2016-05-18 00:57:40,140 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533059883_0001_00_000002, launchTime=1463533060139
2016-05-18 00:57:40,140 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533059883_0001_00_000002
2016-05-18 00:57:40,141 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533059883_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533059883_0001_1_01_000000_0
2016-05-18 00:57:40,141 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533059883_0001_1_01_000000_0] started. Is using containerId: [container_1463533059883_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:40 Starting to run new task attempt: attempt_1463533059883_0001_1_01_000000_0
2016-05-18 00:57:40,141 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533059883_0001_1_01_000000_0
2016-05-18 00:57:40,141 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,141 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FC831479FD67477EB1D51EFC28E077F5, taskAttemptId=attempt_1463533059883_0001_1_01_000000_0, startTime=1463533060141, containerId=container_1463533059883_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:40,142 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:40,142 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826] every/(1/1) ...blyplatform/0_UNKNOWN_ALL, VertexName: FC831479FD67477EB1D51EFC28E077F5, VertexParallelism: 1, TaskAttemptID:attempt_1463533059883_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DF574E3E74A1448C814CE3F445904655, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D7654B6BD38646A7B52BD395671868C1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:40,142 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:40,143 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533059883_0001_1_01_000000_0
2016-05-18 00:57:40,143 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533059883_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]
2016-05-18 00:57:40,143 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533059883_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:40,145 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:40,145 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533059883_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:40,145 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533059883_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:40,147 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:40,149 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533059883_0001_00_000001, Exiting
2016-05-18 00:57:40,149 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533059883_0001_00_000001
2016-05-18 00:57:40,150 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533059883_0001_00_000001
2016-05-18 00:57:40,151 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533059883_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:40,152 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533059883_0001_00_000001 completed successfully
2016-05-18 00:57:40,153 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FC831479FD67477EB1D51EFC28E077F5, ordinal: 1
2016-05-18 00:57:40,153 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FC831479FD67477EB1D51EFC28E077F5, mem on start (mb), free: 106, total: 365, max: 672
2016-05-18 00:57:40,154 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:40,154 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:40,154 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:40,155 INFO output.MROutput (MROutput.java:initialize(421)) - D7654B6BD38646A7B52BD395671868C1: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:40,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:40,156 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DF574E3E74A1448C814CE3F445904655:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [D7654B6BD38646A7B52BD395671868C1:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:40,156 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:40,156 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:40,156 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:40,156 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DF574E3E74A1448C814CE3F445904655:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [D7654B6BD38646A7B52BD395671868C1:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:40,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:40,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DF574E3E74A1448C814CE3F445904655 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:40,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:40,156 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:40,157 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - DF574E3E74A1448C814CE3F445904655: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:40,157 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - DF574E3E74A1448C814CE3F445904655: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:40,158 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: DF574E3E74A1448C814CE3F445904655 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:40,158 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DF574E3E74A1448C814CE3F445904655
2016-05-18 00:57:40,159 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:40,159 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533059883_0001_1_01_000000_0
2016-05-18 00:57:40,161 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:40,162 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: E75539C6ABC34BBFA1C07A09130A034A
2016-05-18 00:57:40,162 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_ALL"], id: D7654B6BD38646A7B52BD395671868C1
2016-05-18 00:57:40,162 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DF574E3E74A1448C814CE3F445904655: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:40,162 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_ALL"]
2016-05-18 00:57:40,163 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] E75539C6ABC34BBFA1C07A09130A034A, for 1 inputs
2016-05-18 00:57:40,164 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533059883_0001_1_00_000000_0_10159} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533060164, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:40,164 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : DF574E3E74A1448C814CE3F445904655
2016-05-18 00:57:40,164 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:40,164 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: DF574E3E74A1448C814CE3F445904655, wasInterrupted=false
2016-05-18 00:57:40,168 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:40,168 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - DF574E3E74A1448C814CE3F445904655: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:40,168 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:40,170 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:40,170 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001_wd/localmode-local-dir/output/attempt_1463533059883_0001_1_00_000000_0_10159/file.out, len=181, isLocal=true
2016-05-18 00:57:40,170 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:40,170 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:40,170 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:40,170 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:40,170 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : DF574E3E74A1448C814CE3F445904655
2016-05-18 00:57:40,170 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - DF574E3E74A1448C814CE3F445904655: Shuffle Runner thread complete
2016-05-18 00:57:40,170 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FC831479FD67477EB1D51EFC28E077F5, all 1 inputs ready in: 00:00:00.007
2016-05-18 00:57:40,172 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:40,172 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - DF574E3E74A1448C814CE3F445904655: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:40,176 INFO output.MROutput (MROutput.java:close(526)) - D7654B6BD38646A7B52BD395671868C1 closed
2016-05-18 00:57:40,177 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533059883_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:40,177 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533059883_0001_1_01_000000_0
2016-05-18 00:57:40,178 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FC831479FD67477EB1D51EFC28E077F5, mem on close (mb), free: 101, total: 365, max: 672
2016-05-18 00:57:40,178 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: DF574E3E74A1448C814CE3F445904655
2016-05-18 00:57:40,178 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DF574E3E74A1448C814CE3F445904655: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:40,178 INFO output.MROutput (MROutput.java:close(526)) - D7654B6BD38646A7B52BD395671868C1 closed
2016-05-18 00:57:40,178 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:40 Completed running task attempt: attempt_1463533059883_0001_1_01_000000_0
2016-05-18 00:57:40,179 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533059883_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:40,179 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533059883_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:40,180 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533059883_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533060171, Process_Duration=6, Process_End_Time=1463533060177, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:40,181 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533059883_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:40,181 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FC831479FD67477EB1D51EFC28E077F5, taskAttemptId=attempt_1463533059883_0001_1_01_000000_0, creationTime=1463533060138, allocationTime=1463533060139, startTime=1463533060141, finishTime=1463533060181, timeTaken=40, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:40,181 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:40,181 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533059883_0001_1_01_000000_0
2016-05-18 00:57:40,181 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:TASK_FINISHED]: vertexName=FC831479FD67477EB1D51EFC28E077F5, taskId=task_1463533059883_0001_1_01_000000, startTime=1463533060141, finishTime=1463533060181, timeTaken=40, status=SUCCEEDED, successfulAttemptID=attempt_1463533059883_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533060171, Process_Duration=6, Process_End_Time=1463533060177, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:40,181 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:40,181 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5]
2016-05-18 00:57:40,181 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:40,182 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,182 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:VERTEX_FINISHED]: vertexName=FC831479FD67477EB1D51EFC28E077F5, vertexId=vertex_1463533059883_0001_1_01, initRequestedTime=1463533060054, initedTime=1463533060058, startRequestedTime=1463533060068, startedTime=1463533060070, finishTime=1463533060181, timeTaken=111, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533060171, Process_Duration=6, Process_End_Time=1463533060177, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533060141, firstTasksToStart=[ task_1463533059883_0001_1_01_000000 ], lastTaskFinishTime=1463533060181, lastTasksToFinish=[ task_1463533059883_0001_1_01_000000 ], minTaskDuration=40, maxTaskDuration=40, avgTaskDuration=40.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533059883_0001_1_01_000000 ], longestDurationTasks=[ task_1463533059883_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:40,182 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:40,182 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:40,182 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:40,182 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533059883_0001_1_00 [DF574E3E74A1448C814CE3F445904655]
2016-05-18 00:57:40,182 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533059883_0001_1
2016-05-18 00:57:40,182 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533059883_0001_1
2016-05-18 00:57:40,182 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533059883_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:40,183 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533059883_0001_00_000002
2016-05-18 00:57:40,182 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: D7654B6BD38646A7B52BD395671868C1 for vertex: vertex_1463533059883_0001_1_01 [FC831479FD67477EB1D51EFC28E077F5], outputName: D7654B6BD38646A7B52BD395671868C1
2016-05-18 00:57:40,183 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533059883_0001_00_000002, stoppedTime=1463533060183, exitStatus=0
2016-05-18 00:57:40,184 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533059883_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:40,184 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533059883_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:40,184 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533059883_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:40,184 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533059883_0001_00_000002, Exiting
2016-05-18 00:57:40,184 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533059883_0001_00_000002
2016-05-18 00:57:40,184 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533059883_0001_00_000002
2016-05-18 00:57:40,184 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533059883_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:40,184 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533059883_0001_00_000002 completed successfully
2016-05-18 00:57:40,190 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:D7654B6BD38646A7B52BD395671868C1 of vertex/vertexGroup:FC831479FD67477EB1D51EFC28E077F5 isVertexGroupOutput:false
2016-05-18 00:57:40,190 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:40,192 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533059883_0001_1, queueSize=0
2016-05-18 00:57:40,192 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533059883_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533059883_0001_1, startTime=1463533060049, finishTime=1463533060190, timeTaken=141, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=230, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=13, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066120291, Process_Duration=8, Process_End_Time=2927066120299, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:40,192 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533059883_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:40,192 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533059883_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:40 Completed Dag: dag_1463533059883_0001_1
2016-05-18 00:57:40,193 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533059883_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:40,193 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533059883_0001_1_post
2016-05-18 00:57:40,193 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,193 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:40,193 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:40,193 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826] every/(1/1) ...blyplatform/0_UNKNOWN_ALL, with id=dag_1463533059883_0001_1
2016-05-18 00:57:40,195 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826] every/(1/1) ...blyplatform/0_UNKNOWN_ALL, with id=dag_1463533059883_0001_1
2016-05-18 00:57:40,201 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...blyplatform/0_UNKNOWN_ALL, applicationId=application_1463533059883_0001
2016-05-18 00:57:40,202 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:40,203 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:40,203 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:40,203 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:40,203 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:40,203 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:40,203 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:40,204 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:40,204 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:40,204 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:40,205 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:40,205 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:40,205 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:40,206 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:40,209 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/757B5282D7304B1988EF8B4FA7C527EF/D4EEFE940B4C40B8B14CBC898A1E9826/.tez/application_1463533059883_0001
2016-05-18 00:57:40,210 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_ALL/_temporary
2016-05-18 00:57:40,210 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.346
2016-05-18 00:57:40,291 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_UNKNOWN_ALL
2016-05-18 00:57:40,313 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:40,314 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:40,345 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 00:57:40,353 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.015
2016-05-18 00:57:40,353 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:40,354 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:40,354 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:40,368 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:40,373 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_REPLACE
2016-05-18 00:57:40,419 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:40,419 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:40,419 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_REPLACE"]
2016-05-18 00:57:40,420 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:40,420 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:40,420 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:40,421 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...latform/0_UNKNOWN_REPLACE
2016-05-18 00:57:40,421 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:40,434 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:40,438 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:40,439 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:40,439 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:40,447 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001 doesn't exist and is created
2016-05-18 00:57:40,461 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:40,462 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:40,462 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001_wd
2016-05-18 00:57:40,471 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533060439_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:40,471 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:40,472 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:40,473 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:40,473 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:40,473 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:40,473 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:40,474 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:40,474 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:40,474 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:40,475 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:40,475 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:40,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:40,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:40,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:40,476 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:40,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:40,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:40,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:40,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:40,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:40,477 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:40,478 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:40,479 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:40,479 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:40,479 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001_wd/localmode-log-dir
2016-05-18 00:57:40,479 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001_wd/localmode-log-dir/history.txt.appattempt_1463533060439_0001_000000, maxErrors=10
2016-05-18 00:57:40,480 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:40,480 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533060439_0001_000000, appSubmitTime=1463533060471, launchTime=1463533060471
2016-05-18 00:57:40,482 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:40,487 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:53599
2016-05-18 00:57:40,488 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533060439
2016-05-18 00:57:40,489 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533060439_0001_000000, startTime=1463533060489
2016-05-18 00:57:40,490 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:40,562 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:40,562 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:40,562 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...latform/0_UNKNOWN_REPLACE, applicationId=application_1463533060439_0001, dagName=[D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8] every/(1/1) ...latform/0_UNKNOWN_REPLACE
2016-05-18 00:57:40,577 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8] every/(1/1) ...latform/0_UNKNOWN_REPLACE
2016-05-18 00:57:40,577 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:40,578 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533060439_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001_wd/localmode-log-dir/dag_1463533060439_0001_1.dot
2016-05-18 00:57:40,578 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001_wd/localmode-log-dir/dag_1463533060439_0001_1-tez-dag.pb.txt
2016-05-18 00:57:40,584 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533060439_0001_1
2016-05-18 00:57:40,584 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,585 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8] every/(1/1) ...latform/0_UNKNOWN_REPLACE
2016-05-18 00:57:40 Running Dag: dag_1463533060439_0001_1
2016-05-18 00:57:40,593 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533060439_0001_1, submitTime=1463533060577
2016-05-18 00:57:40,594 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001/tez-conf.pb" } size: 36913 timestamp: 1463533060000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:40,594 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,594 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:40,595 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001/tez-conf.pb" } size: 36913 timestamp: 1463533060000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:40,595 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,595 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:40,595 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:40,596 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533060439_0001_1, initTime=1463533060595
2016-05-18 00:57:40,596 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533060439_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:40,596 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:40,596 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:DAG_STARTED]: dagID=dag_1463533060439_0001_1, startTime=1463533060596
2016-05-18 00:57:40,596 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533060439_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:40,597 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A56708104DEE465EBF46F5BEF1516872 : {D9839A1F19EC4DA787832753485B52D7={InputName=D9839A1F19EC4DA787832753485B52D7}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:40,597 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: D9839A1F19EC4DA787832753485B52D7, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:40,597 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,597 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,597 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,597 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,598 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:40,598 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,598 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: D9839A1F19EC4DA787832753485B52D7 on vertex vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,598 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...latform/0_UNKNOWN_REPLACE, applicationId=application_1463533060439_0001, dagName=[D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8] every/(1/1) ...latform/0_UNKNOWN_REPLACE
2016-05-18 00:57:40,600 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input D9839A1F19EC4DA787832753485B52D7 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:40,601 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:40,601 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,601 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,602 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533060439_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:40,602 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9], numAdditionalOutputs=1
2016-05-18 00:57:40,602 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=81CEABD18AF84BA8BA478F3926316976, vertex=vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:40,602 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:40,603 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:40,603 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:40,603 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:40,603 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: D9839A1F19EC4DA787832753485B52D7 on vertex vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,604 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for AB100905AB21453C92DD2441F28D67F9:81CEABD18AF84BA8BA478F3926316976 using oldmapred API
2016-05-18 00:57:40,604 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=81CEABD18AF84BA8BA478F3926316976, vertexName=AB100905AB21453C92DD2441F28D67F9, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:40,605 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AB100905AB21453C92DD2441F28D67F9, vertexId=vertex_1463533060439_0001_1_01, initRequestedTime=1463533060598, initedTime=1463533060602, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:40,606 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:40,606 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:40,606 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {D9839A1F19EC4DA787832753485B52D7=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:40,607 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] parallelism set to 1
2016-05-18 00:57:40,607 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:40,607 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A56708104DEE465EBF46F5BEF1516872, vertexId=vertex_1463533060439_0001_1_00, initRequestedTime=1463533060597, initedTime=1463533060607, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:40,608 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] with distanceFromRoot: 0
2016-05-18 00:57:40,608 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:40,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533060439_0001_1_00, startRequestedTime=1463533060598, startedTime=1463533060608
2016-05-18 00:57:40,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533060439_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:40,608 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:40,608 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533060439_0001_1_00 for vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] numStartedSources: 1 numSources: 1
2016-05-18 00:57:40,608 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] with distanceFromRoot: 1
2016-05-18 00:57:40,609 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533060439_0001_1_01, startRequestedTime=1463533060608, startedTime=1463533060608
2016-05-18 00:57:40,609 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:40,612 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: AB100905AB21453C92DD2441F28D67F9 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:40,613 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A56708104DEE465EBF46F5BEF1516872
2016-05-18 00:57:40,613 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A56708104DEE465EBF46F5BEF1516872 in vertex: AB100905AB21453C92DD2441F28D67F9 numBipartiteSourceTasks: 1
2016-05-18 00:57:40,613 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,614 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:TASK_STARTED]: vertexName=A56708104DEE465EBF46F5BEF1516872, taskId=task_1463533060439_0001_1_00_000000, scheduledTime=1463533060613, launchTime=1463533060613
2016-05-18 00:57:40,614 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,615 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:40,616 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533060439_0001_00_000001, launchTime=1463533060616
2016-05-18 00:57:40,617 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533060439_0001_00_000001
2016-05-18 00:57:40,617 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533060439_0001_1_00_000000_0] started. Is using containerId: [container_1463533060439_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:40,617 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A56708104DEE465EBF46F5BEF1516872, taskAttemptId=attempt_1463533060439_0001_1_00_000000_0, startTime=1463533060617, containerId=container_1463533060439_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:40,618 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533060439_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533060439_0001_1_00_000000_0
2016-05-18 00:57:40 Starting to run new task attempt: attempt_1463533060439_0001_1_00_000000_0
2016-05-18 00:57:40,618 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533060439_0001_1_00_000000_0
2016-05-18 00:57:40,619 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,619 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:40,620 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8] every/(1/1) ...latform/0_UNKNOWN_REPLACE, VertexName: A56708104DEE465EBF46F5BEF1516872, VertexParallelism: 1, TaskAttemptID:attempt_1463533060439_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D9839A1F19EC4DA787832753485B52D7, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=AB100905AB21453C92DD2441F28D67F9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:40,620 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:40,621 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533060439_0001_1_00_000000_0
2016-05-18 00:57:40,622 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533060439_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:40,622 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533060439_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:40,623 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533060439_0001, with dag id: dag_1463533060439_0001_1
2016-05-18 00:57:40,626 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:40,629 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A56708104DEE465EBF46F5BEF1516872, ordinal: 0
2016-05-18 00:57:40,629 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A56708104DEE465EBF46F5BEF1516872, mem on start (mb), free: 198, total: 366, max: 672
2016-05-18 00:57:40,629 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:40,630 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:40,630 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:40,630 INFO input.MRInput (MRInput.java:initialize(440)) - D9839A1F19EC4DA787832753485B52D7 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:40,637 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: D9839A1F19EC4DA787832753485B52D7
2016-05-18 00:57:40,637 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:40,637 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D9839A1F19EC4DA787832753485B52D7:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [AB100905AB21453C92DD2441F28D67F9:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:40,637 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:40,638 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:40,638 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:40,638 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D9839A1F19EC4DA787832753485B52D7:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [AB100905AB21453C92DD2441F28D67F9:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:40,638 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:40,638 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D9839A1F19EC4DA787832753485B52D7 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:40,638 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D9839A1F19EC4DA787832753485B52D7
2016-05-18 00:57:40,639 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:40,641 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:40,641 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:40,642 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533060439_0001_1_00_000000_0
2016-05-18 00:57:40,646 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:40,646 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: D9839A1F19EC4DA787832753485B52D7
2016-05-18 00:57:40,646 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 15A9BA714B31487E814E20C98B5A26E3
2016-05-18 00:57:40,647 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 15A9BA714B31487E814E20C98B5A26E3
2016-05-18 00:57:40,648 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:40,649 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:40,649 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - D9839A1F19EC4DA787832753485B52D7 initialized RecordReader from event
2016-05-18 00:57:40,651 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:40,651 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - AB100905AB21453C92DD2441F28D67F9 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@1e9b96b7, comparator=cascading.tuple.hadoop.util.TupleComparator@78c3d813, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:40,652 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for AB100905AB21453C92DD2441F28D67F9: , UsingHashComparator=false
2016-05-18 00:57:40,664 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:40,665 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:40,665 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for AB100905AB21453C92DD2441F28D67F9: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:40,665 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - AB100905AB21453C92DD2441F28D67F9: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:40,666 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:40,666 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A56708104DEE465EBF46F5BEF1516872, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:40,669 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533060439_0001_1_00_000000_0
2016-05-18 00:57:40,669 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A56708104DEE465EBF46F5BEF1516872, mem on close (mb), free: 95, total: 366, max: 672
2016-05-18 00:57:40,669 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - AB100905AB21453C92DD2441F28D67F9: Starting flush of map output
2016-05-18 00:57:40,669 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - AB100905AB21453C92DD2441F28D67F9: Span0.length = 20, perItem = 6
2016-05-18 00:57:40,669 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - AB100905AB21453C92DD2441F28D67F9: done sorting span=0, length=20, time=0
2016-05-18 00:57:40,670 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - AB100905AB21453C92DD2441F28D67F9: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:40,671 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - AB100905AB21453C92DD2441F28D67F9: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001_wd/localmode-local-dir/output/attempt_1463533060439_0001_1_00_000000_0_10165_0/file.out
2016-05-18 00:57:40,675 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:40 Completed running task attempt: attempt_1463533060439_0001_1_00_000000_0
2016-05-18 00:57:40,676 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533060439_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:40,676 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533060439_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:40,676 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533060439_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533060666, Process_Duration=3, Process_End_Time=1463533060669, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:40,677 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533060439_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:40,677 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:40,677 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A56708104DEE465EBF46F5BEF1516872, taskAttemptId=attempt_1463533060439_0001_1_00_000000_0, creationTime=1463533060614, allocationTime=1463533060615, startTime=1463533060617, finishTime=1463533060677, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:40,677 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:40,677 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,678 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533060439_0001_1_00_000000_0
2016-05-18 00:57:40,678 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:TASK_FINISHED]: vertexName=A56708104DEE465EBF46F5BEF1516872, taskId=task_1463533060439_0001_1_00_000000, startTime=1463533060617, finishTime=1463533060678, timeTaken=61, status=SUCCEEDED, successfulAttemptID=attempt_1463533060439_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533060666, Process_Duration=3, Process_End_Time=1463533060669, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:40,678 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533060439_0001_00_000001
2016-05-18 00:57:40,679 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:40,679 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872]
2016-05-18 00:57:40,680 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:VERTEX_FINISHED]: vertexName=A56708104DEE465EBF46F5BEF1516872, vertexId=vertex_1463533060439_0001_1_00, initRequestedTime=1463533060597, initedTime=1463533060607, startRequestedTime=1463533060598, startedTime=1463533060608, finishTime=1463533060679, timeTaken=71, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533060666, Process_Duration=3, Process_End_Time=1463533060669, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533060617, firstTasksToStart=[ task_1463533060439_0001_1_00_000000 ], lastTaskFinishTime=1463533060677, lastTasksToFinish=[ task_1463533060439_0001_1_00_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533060439_0001_1_00_000000 ], longestDurationTasks=[ task_1463533060439_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:40,680 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:40,680 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] attempt: attempt_1463533060439_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:40,680 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AB100905AB21453C92DD2441F28D67F9
2016-05-18 00:57:40,680 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533060439_0001_00_000001, stoppedTime=1463533060680, exitStatus=0
2016-05-18 00:57:40,680 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A56708104DEE465EBF46F5BEF1516872, destinationVertexName=AB100905AB21453C92DD2441F28D67F9. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:40,681 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,681 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533060439_0001_1_00 [A56708104DEE465EBF46F5BEF1516872] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:40,681 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:40,681 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533060439_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:40,681 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:TASK_STARTED]: vertexName=AB100905AB21453C92DD2441F28D67F9, taskId=task_1463533060439_0001_1_01_000000, scheduledTime=1463533060681, launchTime=1463533060681
2016-05-18 00:57:40,682 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,683 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533060439_0001_00_000002, launchTime=1463533060683
2016-05-18 00:57:40,683 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533060439_0001_00_000002
2016-05-18 00:57:40,684 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533060439_0001_1_01_000000_0] started. Is using containerId: [container_1463533060439_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:40,684 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AB100905AB21453C92DD2441F28D67F9, taskAttemptId=attempt_1463533060439_0001_1_01_000000_0, startTime=1463533060684, containerId=container_1463533060439_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:40,684 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533060439_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533060439_0001_1_01_000000_0
2016-05-18 00:57:40 Starting to run new task attempt: attempt_1463533060439_0001_1_01_000000_0
2016-05-18 00:57:40,685 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533060439_0001_1_01_000000_0
2016-05-18 00:57:40,697 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,697 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:40,698 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533060439_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:40,698 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533060439_0001_00_000001: 19 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:40,698 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8] every/(1/1) ...latform/0_UNKNOWN_REPLACE, VertexName: AB100905AB21453C92DD2441F28D67F9, VertexParallelism: 1, TaskAttemptID:attempt_1463533060439_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A56708104DEE465EBF46F5BEF1516872, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=81CEABD18AF84BA8BA478F3926316976, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:40,698 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533060439_0001_00_000001, Exiting
2016-05-18 00:57:40,698 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533060439_0001_00_000001
2016-05-18 00:57:40,698 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533060439_0001_00_000001
2016-05-18 00:57:40,698 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:40,699 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533060439_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:40,700 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533060439_0001_00_000001 completed successfully
2016-05-18 00:57:40,700 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533060439_0001_1_01_000000_0
2016-05-18 00:57:40,703 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:40,705 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:40,705 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533060439_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,706 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533060439_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:40,709 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AB100905AB21453C92DD2441F28D67F9, ordinal: 1
2016-05-18 00:57:40,709 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AB100905AB21453C92DD2441F28D67F9, mem on start (mb), free: 299, total: 365, max: 672
2016-05-18 00:57:40,709 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:40,709 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:40,709 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:40,712 INFO output.MROutput (MROutput.java:initialize(421)) - 81CEABD18AF84BA8BA478F3926316976: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:40,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:40,712 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[81CEABD18AF84BA8BA478F3926316976:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A56708104DEE465EBF46F5BEF1516872:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:40,713 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:40,713 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:40,713 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:40,713 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[81CEABD18AF84BA8BA478F3926316976:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A56708104DEE465EBF46F5BEF1516872:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:40,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:40,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A56708104DEE465EBF46F5BEF1516872 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:40,714 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A56708104DEE465EBF46F5BEF1516872: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:40,715 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A56708104DEE465EBF46F5BEF1516872: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:40,716 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A56708104DEE465EBF46F5BEF1516872 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:40,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A56708104DEE465EBF46F5BEF1516872
2016-05-18 00:57:40,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:40,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:40,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:40,718 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533060439_0001_1_01_000000_0
2016-05-18 00:57:40,719 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A56708104DEE465EBF46F5BEF1516872: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:40,721 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:40,722 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 15A9BA714B31487E814E20C98B5A26E3
2016-05-18 00:57:40,722 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_REPLACE"], id: 81CEABD18AF84BA8BA478F3926316976
2016-05-18 00:57:40,722 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_REPLACE"]
2016-05-18 00:57:40,722 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 15A9BA714B31487E814E20C98B5A26E3, for 1 inputs
2016-05-18 00:57:40,724 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533060439_0001_1_00_000000_0_10165} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533060724, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:40,724 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A56708104DEE465EBF46F5BEF1516872
2016-05-18 00:57:40,724 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:40,725 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A56708104DEE465EBF46F5BEF1516872, wasInterrupted=false
2016-05-18 00:57:40,726 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:40,726 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:40,728 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:40,728 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001_wd/localmode-local-dir/output/attempt_1463533060439_0001_1_00_000000_0_10165/file.out, len=181, isLocal=true
2016-05-18 00:57:40,728 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:40,728 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:40,728 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:40,729 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:40,729 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A56708104DEE465EBF46F5BEF1516872
2016-05-18 00:57:40,729 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AB100905AB21453C92DD2441F28D67F9, all 1 inputs ready in: 00:00:00.007
2016-05-18 00:57:40,729 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A56708104DEE465EBF46F5BEF1516872: Shuffle Runner thread complete
2016-05-18 00:57:40,730 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:40,730 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A56708104DEE465EBF46F5BEF1516872: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:40,732 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:40,734 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:40,737 INFO output.MROutput (MROutput.java:close(526)) - 81CEABD18AF84BA8BA478F3926316976 closed
2016-05-18 00:57:40,737 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533060439_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:40,737 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533060439_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:40,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533060439_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533060729, Process_Duration=8, Process_End_Time=1463533060737, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:40,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:40,738 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AB100905AB21453C92DD2441F28D67F9, mem on close (mb), free: 295, total: 365, max: 672
2016-05-18 00:57:40,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=AB100905AB21453C92DD2441F28D67F9, index=1, interruptedStatus=false
2016-05-18 00:57:40,739 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A56708104DEE465EBF46F5BEF1516872
2016-05-18 00:57:40,739 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A56708104DEE465EBF46F5BEF1516872: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:40,739 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=AB100905AB21453C92DD2441F28D67F9, sourceVertex=A56708104DEE465EBF46F5BEF1516872, interruptedStatus=true
2016-05-18 00:57:40,739 INFO output.MROutput (MROutput.java:close(526)) - 81CEABD18AF84BA8BA478F3926316976 closed
2016-05-18 00:57:40,739 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=AB100905AB21453C92DD2441F28D67F9, sourceVertex=81CEABD18AF84BA8BA478F3926316976, interruptedStatus=true
2016-05-18 00:57:40,740 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533060439_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], containerShutdownRequested=false}
2016-05-18 00:57:40,740 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], containerShutdownRequested=false}
2016-05-18 00:57:40,740 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AB100905AB21453C92DD2441F28D67F9, taskAttemptId=attempt_1463533060439_0001_1_01_000000_0, creationTime=1463533060681, allocationTime=1463533060682, startTime=1463533060684, finishTime=1463533060740, timeTaken=56, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533060439_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=14, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533060729, Process_Duration=8, Process_End_Time=1463533060737, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:40,741 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533060439_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:40,741 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:TASK_FINISHED]: vertexName=AB100905AB21453C92DD2441F28D67F9, taskId=task_1463533060439_0001_1_01_000000, startTime=1463533060684, finishTime=1463533060741, timeTaken=57, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533060439_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:40,741 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] because task failed: task_1463533060439_0001_1_01_000000
2016-05-18 00:57:40,741 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:40,741 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:40,742 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:40,742 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,740 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533060439_0001_00_000002
2016-05-18 00:57:40,742 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=81CEABD18AF84BA8BA478F3926316976, vertexId=vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,742 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533060439_0001_00_000002
2016-05-18 00:57:40,742 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533060439_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]'}
2016-05-18 00:57:40,742 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533060439_0001_00_000002 completed but with errors
2016-05-18 00:57:40,743 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:VERTEX_FINISHED]: vertexName=AB100905AB21453C92DD2441F28D67F9, vertexId=vertex_1463533060439_0001_1_01, initRequestedTime=1463533060598, initedTime=1463533060602, startRequestedTime=1463533060608, startedTime=1463533060608, finishTime=1463533060743, timeTaken=135, status=FAILED, diagnostics=Task failed, taskId=task_1463533060439_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533060439_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533060684, firstTasksToStart=[ task_1463533060439_0001_1_01_000000 ], lastTaskFinishTime=1463533060740, lastTasksToFinish=[ task_1463533060439_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:40,743 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:40,744 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533060439_0001_00_000002, stoppedTime=1463533060744, exitStatus=-1000
2016-05-18 00:57:40,744 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533060439_0001_1_01_000000_0 failed: true container: container_1463533060439_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:40,744 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533060439_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
2016-05-18 00:57:40,744 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:40,744 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:40,744 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:40,746 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533060439_0001_1, queueSize=0
2016-05-18 00:57:40,746 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060439_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533060439_0001_1, startTime=1463533060596, finishTime=1463533060745, timeTaken=149, status=FAILED, diagnostics=Vertex failed, vertexName=AB100905AB21453C92DD2441F28D67F9, vertexId=vertex_1463533060439_0001_1_01, diagnostics=[Task failed, taskId=task_1463533060439_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533060439_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=330, AM_GC_TIME_MILLIS=11, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533060666, Process_Duration=3, Process_End_Time=1463533060669, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:40,750 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533060439_0001_1_01 [AB100905AB21453C92DD2441F28D67F9]
2016-05-18 00:57:40,751 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533060439_0001_1 finished with state: FAILED
2016-05-18 00:57:40,751 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533060439_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:40,751 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533060439_0001_00_000002
2016-05-18 00:57:40 Completed Dag: dag_1463533060439_0001_1
2016-05-18 00:57:40,753 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533060439_0001_1, dagState=FAILED
2016-05-18 00:57:40,754 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533060439_0001_1_post
2016-05-18 00:57:40,755 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:40,755 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:40,755 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:40,755 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8] every/(1/1) ...latform/0_UNKNOWN_REPLACE, with id=dag_1463533060439_0001_1
2016-05-18 00:57:40,756 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8] every/(1/1) ...latform/0_UNKNOWN_REPLACE, with id=dag_1463533060439_0001_1
2016-05-18 00:57:40,756 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...latform/0_UNKNOWN_REPLACE, applicationId=application_1463533060439_0001
2016-05-18 00:57:40,758 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:40,758 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:40,758 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:40,758 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:40,759 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:40,759 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:40,759 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:40,760 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:40,760 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:40,760 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:40,760 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:40,760 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:40,761 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:40,761 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:40,764 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D8BDD31C50BD4A6EB9A7F7B897057434/E9E6035CCD0D4DC89E081A54501F8DC8/.tez/application_1463533060439_0001
2016-05-18 00:57:40,764 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:40,765 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...latform/0_UNKNOWN_REPLACE
2016-05-18 00:57:40,765 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:40,765 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_REPLACE/_temporary
2016-05-18 00:57:40,766 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.346
2016-05-18 00:57:40,766 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_UNKNOWN_REPLACE
2016-05-18 00:57:40,790 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:40,791 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:40,825 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.009
2016-05-18 00:57:40,832 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 00:57:40,833 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:40,833 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:40,833 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:40,854 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:40,862 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_SWAP
2016-05-18 00:57:40,910 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:40,911 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:40,911 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_SWAP"]
2016-05-18 00:57:40,911 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:40,911 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:40,911 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:40,912 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/0_UNKNOWN_SWAP
2016-05-18 00:57:40,912 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:40,924 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:40,928 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:40,928 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:40,928 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:40,937 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001 doesn't exist and is created
2016-05-18 00:57:40,952 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:40,952 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:40,953 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001_wd
2016-05-18 00:57:40,962 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533060928_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:40,962 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:40,963 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:40,964 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:40,964 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:40,964 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:40,964 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:40,965 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:40,967 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:40,967 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:40,967 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:40,967 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:40,967 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:40,967 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:40,967 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:40,968 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:40,968 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:40,968 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:40,968 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:40,968 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:40,970 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:40,973 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:40,973 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:40,973 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001_wd/localmode-log-dir
2016-05-18 00:57:40,973 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001_wd/localmode-log-dir/history.txt.appattempt_1463533060928_0001_000000, maxErrors=10
2016-05-18 00:57:40,973 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:40,974 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533060928_0001_000000, appSubmitTime=1463533060962, launchTime=1463533060962
2016-05-18 00:57:40,976 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:40,985 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:36080
2016-05-18 00:57:40,985 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533060928
2016-05-18 00:57:40,987 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533060928_0001_000000, startTime=1463533060987
2016-05-18 00:57:40,987 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:41,026 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:41,026 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:41,052 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:41,052 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:41,053 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/0_UNKNOWN_SWAP, applicationId=application_1463533060928_0001, dagName=[AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722] every/(1/1) ...lyplatform/0_UNKNOWN_SWAP
2016-05-18 00:57:41,069 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722] every/(1/1) ...lyplatform/0_UNKNOWN_SWAP
2016-05-18 00:57:41,069 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:41,070 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533060928_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001_wd/localmode-log-dir/dag_1463533060928_0001_1.dot
2016-05-18 00:57:41,071 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001_wd/localmode-log-dir/dag_1463533060928_0001_1-tez-dag.pb.txt
2016-05-18 00:57:41,078 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533060928_0001_1
2016-05-18 00:57:41,078 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,079 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722] every/(1/1) ...lyplatform/0_UNKNOWN_SWAP
2016-05-18 00:57:41 Running Dag: dag_1463533060928_0001_1
2016-05-18 00:57:41,089 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533060928_0001_1, submitTime=1463533061069
2016-05-18 00:57:41,090 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001/tez-conf.pb" } size: 36913 timestamp: 1463533060000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:41,090 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,090 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:41,091 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001/tez-conf.pb" } size: 36913 timestamp: 1463533060000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:41,091 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]
2016-05-18 00:57:41,091 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:41,091 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:41,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533060928_0001_1, initTime=1463533061091
2016-05-18 00:57:41,092 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533060928_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:41,092 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:41,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:DAG_STARTED]: dagID=dag_1463533060928_0001_1, startTime=1463533061092
2016-05-18 00:57:41,092 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533060928_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:41,092 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A42D76F457D343F69813E8966D00746C : {BABE3D87F9174D46A594897E81EDC194={InputName=BABE3D87F9174D46A594897E81EDC194}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:41,093 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: BABE3D87F9174D46A594897E81EDC194, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:41,093 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,093 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,093 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/0_UNKNOWN_SWAP, applicationId=application_1463533060928_0001, dagName=[AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722] every/(1/1) ...lyplatform/0_UNKNOWN_SWAP
2016-05-18 00:57:41,093 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,094 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,096 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:41,096 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]
2016-05-18 00:57:41,096 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: BABE3D87F9174D46A594897E81EDC194 on vertex vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,098 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input BABE3D87F9174D46A594897E81EDC194 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:41,099 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:41,099 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]
2016-05-18 00:57:41,099 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]
2016-05-18 00:57:41,100 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533060928_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:41,100 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB], numAdditionalOutputs=1
2016-05-18 00:57:41,100 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=393DCB1F39D34C69BD26CCD55A1F8245, vertex=vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:41,101 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:41,101 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:41,102 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:41,102 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:41,103 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for AFCA2913FE4D4804969F466D3A8FB8AB:393DCB1F39D34C69BD26CCD55A1F8245 using oldmapred API
2016-05-18 00:57:41,103 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=393DCB1F39D34C69BD26CCD55A1F8245, vertexName=AFCA2913FE4D4804969F466D3A8FB8AB, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:41,103 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: BABE3D87F9174D46A594897E81EDC194 on vertex vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,104 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AFCA2913FE4D4804969F466D3A8FB8AB, vertexId=vertex_1463533060928_0001_1_01, initRequestedTime=1463533061096, initedTime=1463533061100, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:41,104 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:41,105 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:41,105 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {BABE3D87F9174D46A594897E81EDC194=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:41,105 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] parallelism set to 1
2016-05-18 00:57:41,107 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:41,108 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A42D76F457D343F69813E8966D00746C, vertexId=vertex_1463533060928_0001_1_00, initRequestedTime=1463533061092, initedTime=1463533061107, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:41,108 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] with distanceFromRoot: 0
2016-05-18 00:57:41,108 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:41,109 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533060928_0001_1_00, startRequestedTime=1463533061096, startedTime=1463533061108
2016-05-18 00:57:41,109 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533060928_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:41,109 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:41,110 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533060928_0001_1_00 for vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB] numStartedSources: 1 numSources: 1
2016-05-18 00:57:41,110 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB] with distanceFromRoot: 1
2016-05-18 00:57:41,110 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A42D76F457D343F69813E8966D00746C
2016-05-18 00:57:41,111 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,111 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533060928_0001_1_01, startRequestedTime=1463533061110, startedTime=1463533061111
2016-05-18 00:57:41,112 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:41,112 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:TASK_STARTED]: vertexName=A42D76F457D343F69813E8966D00746C, taskId=task_1463533060928_0001_1_00_000000, scheduledTime=1463533061112, launchTime=1463533061112
2016-05-18 00:57:41,113 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,115 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: AFCA2913FE4D4804969F466D3A8FB8AB with 0 source tasks and 1 pending tasks
2016-05-18 00:57:41,116 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A42D76F457D343F69813E8966D00746C in vertex: AFCA2913FE4D4804969F466D3A8FB8AB numBipartiteSourceTasks: 1
2016-05-18 00:57:41,116 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:41,117 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533060928_0001_00_000001, launchTime=1463533061117
2016-05-18 00:57:41,118 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533060928_0001_00_000001
2016-05-18 00:57:41,119 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533060928_0001_1_00_000000_0] started. Is using containerId: [container_1463533060928_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:41,119 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A42D76F457D343F69813E8966D00746C, taskAttemptId=attempt_1463533060928_0001_1_00_000000_0, startTime=1463533061118, containerId=container_1463533060928_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:41,119 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533060928_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533060928_0001_1_00_000000_0
2016-05-18 00:57:41 Starting to run new task attempt: attempt_1463533060928_0001_1_00_000000_0
2016-05-18 00:57:41,120 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533060928_0001_1_00_000000_0
2016-05-18 00:57:41,120 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,120 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:41,121 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722] every/(1/1) ...lyplatform/0_UNKNOWN_SWAP, VertexName: A42D76F457D343F69813E8966D00746C, VertexParallelism: 1, TaskAttemptID:attempt_1463533060928_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BABE3D87F9174D46A594897E81EDC194, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=AFCA2913FE4D4804969F466D3A8FB8AB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:41,121 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:41,121 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533060928_0001_1_00_000000_0
2016-05-18 00:57:41,122 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533060928_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:41,123 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533060928_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:41,126 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533060928_0001, with dag id: dag_1463533060928_0001_1
2016-05-18 00:57:41,129 INFO input.MRInput (MRInput.java:initialize(440)) - BABE3D87F9174D46A594897E81EDC194 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:41,130 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: BABE3D87F9174D46A594897E81EDC194
2016-05-18 00:57:41,130 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:41,136 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A42D76F457D343F69813E8966D00746C, ordinal: 0
2016-05-18 00:57:41,136 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A42D76F457D343F69813E8966D00746C, mem on start (mb), free: 204, total: 365, max: 672
2016-05-18 00:57:41,137 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:41,137 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:41,137 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:41,137 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:41,137 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BABE3D87F9174D46A594897E81EDC194:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [AFCA2913FE4D4804969F466D3A8FB8AB:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:41,137 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:41,137 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:41,137 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:41,138 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BABE3D87F9174D46A594897E81EDC194:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [AFCA2913FE4D4804969F466D3A8FB8AB:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:41,138 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:41,138 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BABE3D87F9174D46A594897E81EDC194
2016-05-18 00:57:41,138 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BABE3D87F9174D46A594897E81EDC194 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:41,138 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:41,138 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:41,138 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:41,139 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533060928_0001_1_00_000000_0
2016-05-18 00:57:41,140 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:41,140 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - BABE3D87F9174D46A594897E81EDC194 initialized RecordReader from event
2016-05-18 00:57:41,142 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:41,142 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: BABE3D87F9174D46A594897E81EDC194
2016-05-18 00:57:41,142 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: B611AF1CCAAD4DDCA2A332EB95825522
2016-05-18 00:57:41,142 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] B611AF1CCAAD4DDCA2A332EB95825522
2016-05-18 00:57:41,143 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:41,145 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:41,145 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - AFCA2913FE4D4804969F466D3A8FB8AB using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@800e4a9, comparator=cascading.tuple.hadoop.util.TupleComparator@72a41759, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:41,146 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for AFCA2913FE4D4804969F466D3A8FB8AB: , UsingHashComparator=false
2016-05-18 00:57:41,162 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:41,162 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:41,162 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for AFCA2913FE4D4804969F466D3A8FB8AB: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:41,162 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - AFCA2913FE4D4804969F466D3A8FB8AB: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:41,163 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:41,163 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A42D76F457D343F69813E8966D00746C, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:41,165 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533060928_0001_1_00_000000_0
2016-05-18 00:57:41,165 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A42D76F457D343F69813E8966D00746C, mem on close (mb), free: 102, total: 365, max: 672
2016-05-18 00:57:41,166 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - AFCA2913FE4D4804969F466D3A8FB8AB: Starting flush of map output
2016-05-18 00:57:41,166 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - AFCA2913FE4D4804969F466D3A8FB8AB: Span0.length = 20, perItem = 6
2016-05-18 00:57:41,166 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - AFCA2913FE4D4804969F466D3A8FB8AB: done sorting span=0, length=20, time=0
2016-05-18 00:57:41,166 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - AFCA2913FE4D4804969F466D3A8FB8AB: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:41,167 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - AFCA2913FE4D4804969F466D3A8FB8AB: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001_wd/localmode-local-dir/output/attempt_1463533060928_0001_1_00_000000_0_10171_0/file.out
2016-05-18 00:57:41,171 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:41 Completed running task attempt: attempt_1463533060928_0001_1_00_000000_0
2016-05-18 00:57:41,173 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533060928_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:41,173 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533060928_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:41,174 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533060928_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533061163, Process_Duration=2, Process_End_Time=1463533061165, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:41,174 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533060928_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:41,174 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:41,174 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:41,175 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,175 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A42D76F457D343F69813E8966D00746C, taskAttemptId=attempt_1463533060928_0001_1_00_000000_0, creationTime=1463533061113, allocationTime=1463533061116, startTime=1463533061118, finishTime=1463533061175, timeTaken=57, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:41,175 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533060928_0001_1_00_000000_0
2016-05-18 00:57:41,176 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:TASK_FINISHED]: vertexName=A42D76F457D343F69813E8966D00746C, taskId=task_1463533060928_0001_1_00_000000, startTime=1463533061118, finishTime=1463533061175, timeTaken=57, status=SUCCEEDED, successfulAttemptID=attempt_1463533060928_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533061163, Process_Duration=2, Process_End_Time=1463533061165, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:41,177 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:41,177 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,176 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533060928_0001_00_000001
2016-05-18 00:57:41,177 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:VERTEX_FINISHED]: vertexName=A42D76F457D343F69813E8966D00746C, vertexId=vertex_1463533060928_0001_1_00, initRequestedTime=1463533061092, initedTime=1463533061107, startRequestedTime=1463533061096, startedTime=1463533061108, finishTime=1463533061177, timeTaken=69, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533061163, Process_Duration=2, Process_End_Time=1463533061165, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533061118, firstTasksToStart=[ task_1463533060928_0001_1_00_000000 ], lastTaskFinishTime=1463533061175, lastTasksToFinish=[ task_1463533060928_0001_1_00_000000 ], minTaskDuration=57, maxTaskDuration=57, avgTaskDuration=57.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533060928_0001_1_00_000000 ], longestDurationTasks=[ task_1463533060928_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:41,178 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:41,178 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533060928_0001_00_000001, stoppedTime=1463533061178, exitStatus=0
2016-05-18 00:57:41,178 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB] attempt: attempt_1463533060928_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:41,178 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:41,178 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AFCA2913FE4D4804969F466D3A8FB8AB
2016-05-18 00:57:41,178 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A42D76F457D343F69813E8966D00746C, destinationVertexName=AFCA2913FE4D4804969F466D3A8FB8AB. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:41,178 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]
2016-05-18 00:57:41,178 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:41,179 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:TASK_STARTED]: vertexName=AFCA2913FE4D4804969F466D3A8FB8AB, taskId=task_1463533060928_0001_1_01_000000, scheduledTime=1463533061179, launchTime=1463533061179
2016-05-18 00:57:41,179 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533060928_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:41,179 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]
2016-05-18 00:57:41,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533060928_0001_00_000002, launchTime=1463533061180
2016-05-18 00:57:41,181 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533060928_0001_00_000002
2016-05-18 00:57:41,181 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533060928_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533060928_0001_1_01_000000_0
2016-05-18 00:57:41,181 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533060928_0001_1_01_000000_0] started. Is using containerId: [container_1463533060928_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:41,181 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AFCA2913FE4D4804969F466D3A8FB8AB, taskAttemptId=attempt_1463533060928_0001_1_01_000000_0, startTime=1463533061181, containerId=container_1463533060928_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:41 Starting to run new task attempt: attempt_1463533060928_0001_1_01_000000_0
2016-05-18 00:57:41,181 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533060928_0001_1_01_000000_0
2016-05-18 00:57:41,182 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,182 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:41,182 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722] every/(1/1) ...lyplatform/0_UNKNOWN_SWAP, VertexName: AFCA2913FE4D4804969F466D3A8FB8AB, VertexParallelism: 1, TaskAttemptID:attempt_1463533060928_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A42D76F457D343F69813E8966D00746C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=393DCB1F39D34C69BD26CCD55A1F8245, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:41,182 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:41,182 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533060928_0001_1_01_000000_0
2016-05-18 00:57:41,183 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533060928_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]
2016-05-18 00:57:41,183 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533060928_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:41,186 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:41,188 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533060928_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:41,188 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533060928_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:41,188 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533060928_0001_00_000001, Exiting
2016-05-18 00:57:41,188 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533060928_0001_00_000001
2016-05-18 00:57:41,189 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533060928_0001_00_000001
2016-05-18 00:57:41,190 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:41,190 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533060928_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:41,191 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533060928_0001_00_000001 completed successfully
2016-05-18 00:57:41,192 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AFCA2913FE4D4804969F466D3A8FB8AB, ordinal: 1
2016-05-18 00:57:41,192 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AFCA2913FE4D4804969F466D3A8FB8AB, mem on start (mb), free: 97, total: 365, max: 672
2016-05-18 00:57:41,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:41,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:41,193 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:41,194 INFO output.MROutput (MROutput.java:initialize(421)) - 393DCB1F39D34C69BD26CCD55A1F8245: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:41,194 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:41,194 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[393DCB1F39D34C69BD26CCD55A1F8245:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A42D76F457D343F69813E8966D00746C:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:41,194 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:41,195 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:41,195 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:41,195 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[393DCB1F39D34C69BD26CCD55A1F8245:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A42D76F457D343F69813E8966D00746C:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:41,195 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:41,195 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A42D76F457D343F69813E8966D00746C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:41,196 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:41,196 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:41,196 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A42D76F457D343F69813E8966D00746C: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:41,196 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A42D76F457D343F69813E8966D00746C: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:41,197 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A42D76F457D343F69813E8966D00746C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:41,198 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A42D76F457D343F69813E8966D00746C
2016-05-18 00:57:41,198 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:41,198 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533060928_0001_1_01_000000_0
2016-05-18 00:57:41,200 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A42D76F457D343F69813E8966D00746C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:41,201 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:41,202 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533060928_0001_1_00_000000_0_10171} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533061202, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:41,202 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A42D76F457D343F69813E8966D00746C
2016-05-18 00:57:41,202 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:41,202 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A42D76F457D343F69813E8966D00746C, wasInterrupted=false
2016-05-18 00:57:41,202 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:41,203 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: B611AF1CCAAD4DDCA2A332EB95825522
2016-05-18 00:57:41,203 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_SWAP"], id: 393DCB1F39D34C69BD26CCD55A1F8245
2016-05-18 00:57:41,203 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_SWAP"]
2016-05-18 00:57:41,203 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] B611AF1CCAAD4DDCA2A332EB95825522, for 1 inputs
2016-05-18 00:57:41,203 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:41,204 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:41,205 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001_wd/localmode-local-dir/output/attempt_1463533060928_0001_1_00_000000_0_10171/file.out, len=181, isLocal=true
2016-05-18 00:57:41,205 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:41,205 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:41,205 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:41,205 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:41,205 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A42D76F457D343F69813E8966D00746C
2016-05-18 00:57:41,205 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A42D76F457D343F69813E8966D00746C: Shuffle Runner thread complete
2016-05-18 00:57:41,206 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AFCA2913FE4D4804969F466D3A8FB8AB, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:41,207 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:41,208 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A42D76F457D343F69813E8966D00746C: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:41,210 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - A42D76F457D343F69813E8966D00746C: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:41,214 INFO output.MROutput (MROutput.java:close(526)) - 393DCB1F39D34C69BD26CCD55A1F8245 closed
2016-05-18 00:57:41,214 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533060928_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:41,214 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533060928_0001_1_01_000000_0
2016-05-18 00:57:41,215 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AFCA2913FE4D4804969F466D3A8FB8AB, mem on close (mb), free: 94, total: 365, max: 672
2016-05-18 00:57:41,215 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A42D76F457D343F69813E8966D00746C
2016-05-18 00:57:41,215 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A42D76F457D343F69813E8966D00746C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:41,215 INFO output.MROutput (MROutput.java:close(526)) - 393DCB1F39D34C69BD26CCD55A1F8245 closed
2016-05-18 00:57:41,215 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:41 Completed running task attempt: attempt_1463533060928_0001_1_01_000000_0
2016-05-18 00:57:41,216 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533060928_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:41,216 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533060928_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:41,216 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533060928_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533061207, Process_Duration=7, Process_End_Time=1463533061214, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:41,217 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533060928_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:41,217 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:41,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AFCA2913FE4D4804969F466D3A8FB8AB, taskAttemptId=attempt_1463533060928_0001_1_01_000000_0, creationTime=1463533061179, allocationTime=1463533061180, startTime=1463533061181, finishTime=1463533061217, timeTaken=36, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:41,217 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:41,217 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,217 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533060928_0001_1_01_000000_0
2016-05-18 00:57:41,217 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533060928_0001_00_000002
2016-05-18 00:57:41,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:TASK_FINISHED]: vertexName=AFCA2913FE4D4804969F466D3A8FB8AB, taskId=task_1463533060928_0001_1_01_000000, startTime=1463533061181, finishTime=1463533061217, timeTaken=36, status=SUCCEEDED, successfulAttemptID=attempt_1463533060928_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533061207, Process_Duration=7, Process_End_Time=1463533061214, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:41,218 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:41,218 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB]
2016-05-18 00:57:41,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:VERTEX_FINISHED]: vertexName=AFCA2913FE4D4804969F466D3A8FB8AB, vertexId=vertex_1463533060928_0001_1_01, initRequestedTime=1463533061096, initedTime=1463533061100, startRequestedTime=1463533061110, startedTime=1463533061111, finishTime=1463533061218, timeTaken=107, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533061207, Process_Duration=7, Process_End_Time=1463533061214, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533061181, firstTasksToStart=[ task_1463533060928_0001_1_01_000000 ], lastTaskFinishTime=1463533061217, lastTasksToFinish=[ task_1463533060928_0001_1_01_000000 ], minTaskDuration=36, maxTaskDuration=36, avgTaskDuration=36.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533060928_0001_1_01_000000 ], longestDurationTasks=[ task_1463533060928_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:41,219 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:41,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533060928_0001_00_000002, stoppedTime=1463533061219, exitStatus=0
2016-05-18 00:57:41,219 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:41,219 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:41,219 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533060928_0001_1_00 [A42D76F457D343F69813E8966D00746C]
2016-05-18 00:57:41,219 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533060928_0001_1
2016-05-18 00:57:41,220 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533060928_0001_1
2016-05-18 00:57:41,220 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 393DCB1F39D34C69BD26CCD55A1F8245 for vertex: vertex_1463533060928_0001_1_01 [AFCA2913FE4D4804969F466D3A8FB8AB], outputName: 393DCB1F39D34C69BD26CCD55A1F8245
2016-05-18 00:57:41,223 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533060928_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:41,223 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533060928_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:41,226 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:393DCB1F39D34C69BD26CCD55A1F8245 of vertex/vertexGroup:AFCA2913FE4D4804969F466D3A8FB8AB isVertexGroupOutput:false
2016-05-18 00:57:41,226 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:41,228 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533060928_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:41,228 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533060928_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:41,228 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533060928_0001_00_000002, Exiting
2016-05-18 00:57:41,228 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533060928_0001_00_000002
2016-05-18 00:57:41,228 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533060928_0001_00_000002
2016-05-18 00:57:41,229 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533060928_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:41,229 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533060928_0001_00_000002 completed successfully
2016-05-18 00:57:41,229 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533060928_0001_1, queueSize=0
2016-05-18 00:57:41,229 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533060928_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533060928_0001_1, startTime=1463533061092, finishTime=1463533061227, timeTaken=135, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=260, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066122370, Process_Duration=9, Process_End_Time=2927066122379, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:41,229 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533060928_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:41,229 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533060928_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:41 Completed Dag: dag_1463533060928_0001_1
2016-05-18 00:57:41,230 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533060928_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:41,230 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533060928_0001_1_post
2016-05-18 00:57:41,230 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,230 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:41,230 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:41,230 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722] every/(1/1) ...lyplatform/0_UNKNOWN_SWAP, with id=dag_1463533060928_0001_1
2016-05-18 00:57:41,230 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722] every/(1/1) ...lyplatform/0_UNKNOWN_SWAP, with id=dag_1463533060928_0001_1
2016-05-18 00:57:41,240 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/0_UNKNOWN_SWAP, applicationId=application_1463533060928_0001
2016-05-18 00:57:41,242 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:41,242 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:41,242 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:41,242 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:41,242 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:41,243 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:41,243 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:41,244 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:41,244 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:41,245 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:41,245 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:41,245 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:41,246 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:41,247 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:41,251 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/AAFC668162C4423D8FBC0A0DE46BF874/A7BEFB948CF043039A361C5EE9EF3722/.tez/application_1463533060928_0001
2016-05-18 00:57:41,251 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_UNKNOWN_SWAP/_temporary
2016-05-18 00:57:41,252 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.341
2016-05-18 00:57:41,332 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_UNKNOWN_SWAP
2016-05-18 00:57:41,354 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:41,354 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:41,388 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.009
2016-05-18 00:57:41,396 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 00:57:41,397 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:41,397 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:41,397 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:41,411 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:41,416 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_-1
2016-05-18 00:57:41,461 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:41,461 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:41,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_-1"]
2016-05-18 00:57:41,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:41,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:41,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:41,463 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...emblyplatform/0_VALUES_-1
2016-05-18 00:57:41,463 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:41,475 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:41,479 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:41,480 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:41,480 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:41,488 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001 doesn't exist and is created
2016-05-18 00:57:41,501 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:41,501 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:41,502 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001_wd
2016-05-18 00:57:41,513 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533061479_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:41,514 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:41,514 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:41,516 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:41,516 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:41,516 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:41,516 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:41,517 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:41,518 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:41,519 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:41,519 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:41,519 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:41,520 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:41,520 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:41,520 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:41,521 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:41,521 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:41,521 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:41,522 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:41,522 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:41,524 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:41,524 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:41,525 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:41,525 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001_wd/localmode-log-dir
2016-05-18 00:57:41,525 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001_wd/localmode-log-dir/history.txt.appattempt_1463533061479_0001_000000, maxErrors=10
2016-05-18 00:57:41,526 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:41,526 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533061479_0001_000000, appSubmitTime=1463533061513, launchTime=1463533061513
2016-05-18 00:57:41,529 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:41,531 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:33593
2016-05-18 00:57:41,534 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533061479
2016-05-18 00:57:41,535 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533061479_0001_000000, startTime=1463533061535
2016-05-18 00:57:41,535 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:41,602 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:41,602 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:41,602 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...emblyplatform/0_VALUES_-1, applicationId=application_1463533061479_0001, dagName=[E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2] every/(1/1) ...emblyplatform/0_VALUES_-1
2016-05-18 00:57:41,617 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2] every/(1/1) ...emblyplatform/0_VALUES_-1
2016-05-18 00:57:41,617 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:41,618 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533061479_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001_wd/localmode-log-dir/dag_1463533061479_0001_1.dot
2016-05-18 00:57:41,618 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001_wd/localmode-log-dir/dag_1463533061479_0001_1-tez-dag.pb.txt
2016-05-18 00:57:41,624 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533061479_0001_1
2016-05-18 00:57:41,624 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,624 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2] every/(1/1) ...emblyplatform/0_VALUES_-1
2016-05-18 00:57:41 Running Dag: dag_1463533061479_0001_1
2016-05-18 00:57:41,633 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533061479_0001_1, submitTime=1463533061617
2016-05-18 00:57:41,634 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:41,634 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:41,634 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001/tez-conf.pb" } size: 36913 timestamp: 1463533061000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:41,634 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,634 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:41,635 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001/tez-conf.pb" } size: 36913 timestamp: 1463533061000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:41,635 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,635 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:41,635 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:41,636 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533061479_0001_1, initTime=1463533061635
2016-05-18 00:57:41,636 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533061479_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:41,636 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:41,636 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:DAG_STARTED]: dagID=dag_1463533061479_0001_1, startTime=1463533061636
2016-05-18 00:57:41,636 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533061479_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:41,636 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E67DAADF713241B9AE670519629E424C : {C0F739E6ADEC417DB99A2B8FBB820CD2={InputName=C0F739E6ADEC417DB99A2B8FBB820CD2}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:41,636 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: C0F739E6ADEC417DB99A2B8FBB820CD2, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:41,636 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,637 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,637 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,637 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,637 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:41,638 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: C0F739E6ADEC417DB99A2B8FBB820CD2 on vertex vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,638 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...emblyplatform/0_VALUES_-1, applicationId=application_1463533061479_0001, dagName=[E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2] every/(1/1) ...emblyplatform/0_VALUES_-1
2016-05-18 00:57:41,639 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input C0F739E6ADEC417DB99A2B8FBB820CD2 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:41,638 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,641 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:41,641 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:41,641 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:41,641 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:41,641 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: C0F739E6ADEC417DB99A2B8FBB820CD2 on vertex vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,643 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:41,643 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,643 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,644 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533061479_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:41,644 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43], numAdditionalOutputs=1
2016-05-18 00:57:41,644 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=C762E2AC9A5D4070ABC00C5837E758E4, vertex=vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:41,646 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D00A46C5316A4874B0AE33E0BE199B43:C762E2AC9A5D4070ABC00C5837E758E4 using oldmapred API
2016-05-18 00:57:41,646 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=C762E2AC9A5D4070ABC00C5837E758E4, vertexName=D00A46C5316A4874B0AE33E0BE199B43, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:41,647 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D00A46C5316A4874B0AE33E0BE199B43, vertexId=vertex_1463533061479_0001_1_01, initRequestedTime=1463533061638, initedTime=1463533061644, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:41,648 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:41,648 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:41,648 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {C0F739E6ADEC417DB99A2B8FBB820CD2=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:41,649 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] parallelism set to 1
2016-05-18 00:57:41,649 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:41,650 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E67DAADF713241B9AE670519629E424C, vertexId=vertex_1463533061479_0001_1_00, initRequestedTime=1463533061636, initedTime=1463533061650, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:41,650 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] with distanceFromRoot: 0
2016-05-18 00:57:41,650 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:41,651 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533061479_0001_1_00, startRequestedTime=1463533061638, startedTime=1463533061651
2016-05-18 00:57:41,651 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533061479_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:41,652 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:41,652 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533061479_0001_1_00 for vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] numStartedSources: 1 numSources: 1
2016-05-18 00:57:41,652 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] with distanceFromRoot: 1
2016-05-18 00:57:41,652 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E67DAADF713241B9AE670519629E424C
2016-05-18 00:57:41,652 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533061479_0001_1_01, startRequestedTime=1463533061652, startedTime=1463533061652
2016-05-18 00:57:41,653 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:41,653 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,654 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D00A46C5316A4874B0AE33E0BE199B43 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:41,654 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:TASK_STARTED]: vertexName=E67DAADF713241B9AE670519629E424C, taskId=task_1463533061479_0001_1_00_000000, scheduledTime=1463533061654, launchTime=1463533061654
2016-05-18 00:57:41,655 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,656 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E67DAADF713241B9AE670519629E424C in vertex: D00A46C5316A4874B0AE33E0BE199B43 numBipartiteSourceTasks: 1
2016-05-18 00:57:41,657 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:41,658 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533061479_0001_00_000001, launchTime=1463533061658
2016-05-18 00:57:41,659 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533061479_0001_00_000001
2016-05-18 00:57:41,659 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533061479_0001_1_00_000000_0] started. Is using containerId: [container_1463533061479_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:41,659 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E67DAADF713241B9AE670519629E424C, taskAttemptId=attempt_1463533061479_0001_1_00_000000_0, startTime=1463533061659, containerId=container_1463533061479_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:41,660 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533061479_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533061479_0001_1_00_000000_0
2016-05-18 00:57:41 Starting to run new task attempt: attempt_1463533061479_0001_1_00_000000_0
2016-05-18 00:57:41,660 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533061479_0001_1_00_000000_0
2016-05-18 00:57:41,660 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,661 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:41,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2] every/(1/1) ...emblyplatform/0_VALUES_-1, VertexName: E67DAADF713241B9AE670519629E424C, VertexParallelism: 1, TaskAttemptID:attempt_1463533061479_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C0F739E6ADEC417DB99A2B8FBB820CD2, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D00A46C5316A4874B0AE33E0BE199B43, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:41,661 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:41,661 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533061479_0001, with dag id: dag_1463533061479_0001_1
2016-05-18 00:57:41,662 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533061479_0001_1_00_000000_0
2016-05-18 00:57:41,664 INFO input.MRInput (MRInput.java:initialize(440)) - C0F739E6ADEC417DB99A2B8FBB820CD2 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:41,665 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:41,665 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: C0F739E6ADEC417DB99A2B8FBB820CD2
2016-05-18 00:57:41,666 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533061479_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:41,667 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533061479_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:41,669 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E67DAADF713241B9AE670519629E424C, ordinal: 0
2016-05-18 00:57:41,669 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E67DAADF713241B9AE670519629E424C, mem on start (mb), free: 199, total: 365, max: 672
2016-05-18 00:57:41,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:41,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:41,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:41,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:41,669 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[C0F739E6ADEC417DB99A2B8FBB820CD2:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D00A46C5316A4874B0AE33E0BE199B43:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:41,669 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:41,669 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:41,669 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:41,670 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[C0F739E6ADEC417DB99A2B8FBB820CD2:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D00A46C5316A4874B0AE33E0BE199B43:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:41,670 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:41,670 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C0F739E6ADEC417DB99A2B8FBB820CD2
2016-05-18 00:57:41,670 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C0F739E6ADEC417DB99A2B8FBB820CD2 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:41,670 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:41,671 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:41,671 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:41,671 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533061479_0001_1_00_000000_0
2016-05-18 00:57:41,672 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:41,672 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - C0F739E6ADEC417DB99A2B8FBB820CD2 initialized RecordReader from event
2016-05-18 00:57:41,674 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:41,675 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: C0F739E6ADEC417DB99A2B8FBB820CD2
2016-05-18 00:57:41,675 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 8714BDD6E4EA4DF4ADD64B3B464C735B
2016-05-18 00:57:41,675 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 8714BDD6E4EA4DF4ADD64B3B464C735B
2016-05-18 00:57:41,676 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:41,679 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:41,679 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D00A46C5316A4874B0AE33E0BE199B43 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@e1d0e53, comparator=cascading.tuple.hadoop.util.TupleComparator@9d68b01, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:41,680 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D00A46C5316A4874B0AE33E0BE199B43: , UsingHashComparator=false
2016-05-18 00:57:41,692 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:41,692 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:41,693 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D00A46C5316A4874B0AE33E0BE199B43: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:41,693 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D00A46C5316A4874B0AE33E0BE199B43: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:41,693 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:41,693 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E67DAADF713241B9AE670519629E424C, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:41,695 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533061479_0001_1_00_000000_0
2016-05-18 00:57:41,696 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E67DAADF713241B9AE670519629E424C, mem on close (mb), free: 97, total: 365, max: 672
2016-05-18 00:57:41,696 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D00A46C5316A4874B0AE33E0BE199B43: Starting flush of map output
2016-05-18 00:57:41,696 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D00A46C5316A4874B0AE33E0BE199B43: Span0.length = 20, perItem = 6
2016-05-18 00:57:41,696 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D00A46C5316A4874B0AE33E0BE199B43: done sorting span=0, length=20, time=0
2016-05-18 00:57:41,697 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D00A46C5316A4874B0AE33E0BE199B43: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:41,697 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D00A46C5316A4874B0AE33E0BE199B43: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001_wd/localmode-local-dir/output/attempt_1463533061479_0001_1_00_000000_0_10177_0/file.out
2016-05-18 00:57:41,703 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:41 Completed running task attempt: attempt_1463533061479_0001_1_00_000000_0
2016-05-18 00:57:41,703 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533061479_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:41,704 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533061479_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:41,705 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533061479_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533061693, Process_Duration=2, Process_End_Time=1463533061695, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:41,706 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533061479_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:41,706 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:41,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E67DAADF713241B9AE670519629E424C, taskAttemptId=attempt_1463533061479_0001_1_00_000000_0, creationTime=1463533061654, allocationTime=1463533061657, startTime=1463533061659, finishTime=1463533061706, timeTaken=47, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:41,706 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:41,706 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,706 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533061479_0001_1_00_000000_0
2016-05-18 00:57:41,707 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:TASK_FINISHED]: vertexName=E67DAADF713241B9AE670519629E424C, taskId=task_1463533061479_0001_1_00_000000, startTime=1463533061659, finishTime=1463533061706, timeTaken=47, status=SUCCEEDED, successfulAttemptID=attempt_1463533061479_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533061693, Process_Duration=2, Process_End_Time=1463533061695, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:41,707 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533061479_0001_00_000001
2016-05-18 00:57:41,707 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:41,707 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C]
2016-05-18 00:57:41,708 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:VERTEX_FINISHED]: vertexName=E67DAADF713241B9AE670519629E424C, vertexId=vertex_1463533061479_0001_1_00, initRequestedTime=1463533061636, initedTime=1463533061650, startRequestedTime=1463533061638, startedTime=1463533061651, finishTime=1463533061707, timeTaken=56, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533061693, Process_Duration=2, Process_End_Time=1463533061695, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533061659, firstTasksToStart=[ task_1463533061479_0001_1_00_000000 ], lastTaskFinishTime=1463533061706, lastTasksToFinish=[ task_1463533061479_0001_1_00_000000 ], minTaskDuration=47, maxTaskDuration=47, avgTaskDuration=47.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533061479_0001_1_00_000000 ], longestDurationTasks=[ task_1463533061479_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:41,708 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:41,708 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] attempt: attempt_1463533061479_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:41,708 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D00A46C5316A4874B0AE33E0BE199B43
2016-05-18 00:57:41,708 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533061479_0001_00_000001, stoppedTime=1463533061708, exitStatus=0
2016-05-18 00:57:41,708 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533061479_0001_1_00 [E67DAADF713241B9AE670519629E424C] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:41,709 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:41,709 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533061479_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:41,708 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E67DAADF713241B9AE670519629E424C, destinationVertexName=D00A46C5316A4874B0AE33E0BE199B43. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:41,709 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:TASK_STARTED]: vertexName=D00A46C5316A4874B0AE33E0BE199B43, taskId=task_1463533061479_0001_1_01_000000, scheduledTime=1463533061709, launchTime=1463533061709
2016-05-18 00:57:41,710 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,713 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533061479_0001_00_000002, launchTime=1463533061711
2016-05-18 00:57:41,713 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533061479_0001_00_000002
2016-05-18 00:57:41,714 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533061479_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533061479_0001_1_01_000000_0
2016-05-18 00:57:41 Starting to run new task attempt: attempt_1463533061479_0001_1_01_000000_0
2016-05-18 00:57:41,714 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533061479_0001_1_01_000000_0] started. Is using containerId: [container_1463533061479_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:41,714 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533061479_0001_1_01_000000_0
2016-05-18 00:57:41,715 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,715 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:41,715 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D00A46C5316A4874B0AE33E0BE199B43, taskAttemptId=attempt_1463533061479_0001_1_01_000000_0, startTime=1463533061714, containerId=container_1463533061479_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:41,715 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2] every/(1/1) ...emblyplatform/0_VALUES_-1, VertexName: D00A46C5316A4874B0AE33E0BE199B43, VertexParallelism: 1, TaskAttemptID:attempt_1463533061479_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E67DAADF713241B9AE670519629E424C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C762E2AC9A5D4070ABC00C5837E758E4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:41,716 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:41,716 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533061479_0001_1_01_000000_0
2016-05-18 00:57:41,717 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533061479_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,717 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533061479_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:41,717 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533061479_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:41,717 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533061479_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:41,718 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533061479_0001_00_000001, Exiting
2016-05-18 00:57:41,718 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533061479_0001_00_000001
2016-05-18 00:57:41,718 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533061479_0001_00_000001
2016-05-18 00:57:41,719 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:41,720 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:41,720 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533061479_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:41,721 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533061479_0001_00_000001 completed successfully
2016-05-18 00:57:41,725 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D00A46C5316A4874B0AE33E0BE199B43, ordinal: 1
2016-05-18 00:57:41,725 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D00A46C5316A4874B0AE33E0BE199B43, mem on start (mb), free: 92, total: 365, max: 672
2016-05-18 00:57:41,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:41,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:41,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:41,730 INFO output.MROutput (MROutput.java:initialize(421)) - C762E2AC9A5D4070ABC00C5837E758E4: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:41,730 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:41,730 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[E67DAADF713241B9AE670519629E424C:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [C762E2AC9A5D4070ABC00C5837E758E4:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:41,730 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:41,731 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:41,731 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:41,731 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[E67DAADF713241B9AE670519629E424C:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [C762E2AC9A5D4070ABC00C5837E758E4:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:41,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:41,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E67DAADF713241B9AE670519629E424C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:41,732 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E67DAADF713241B9AE670519629E424C: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:41,732 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E67DAADF713241B9AE670519629E424C: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:41,732 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:41,733 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:41,733 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E67DAADF713241B9AE670519629E424C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:41,746 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E67DAADF713241B9AE670519629E424C
2016-05-18 00:57:41,746 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:41,748 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533061479_0001_1_01_000000_0
2016-05-18 00:57:41,748 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E67DAADF713241B9AE670519629E424C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:41,751 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:41,751 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533061479_0001_1_00_000000_0_10177} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533061751, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:41,751 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E67DAADF713241B9AE670519629E424C
2016-05-18 00:57:41,751 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:41,751 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 8714BDD6E4EA4DF4ADD64B3B464C735B
2016-05-18 00:57:41,751 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E67DAADF713241B9AE670519629E424C, wasInterrupted=false
2016-05-18 00:57:41,751 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_-1"], id: C762E2AC9A5D4070ABC00C5837E758E4
2016-05-18 00:57:41,752 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_-1"]
2016-05-18 00:57:41,752 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:41,752 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 8714BDD6E4EA4DF4ADD64B3B464C735B, for 1 inputs
2016-05-18 00:57:41,753 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:41,754 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:41,754 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001_wd/localmode-local-dir/output/attempt_1463533061479_0001_1_00_000000_0_10177/file.out, len=181, isLocal=true
2016-05-18 00:57:41,754 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:41,754 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:41,754 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:41,754 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:41,755 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E67DAADF713241B9AE670519629E424C
2016-05-18 00:57:41,755 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E67DAADF713241B9AE670519629E424C: Shuffle Runner thread complete
2016-05-18 00:57:41,755 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D00A46C5316A4874B0AE33E0BE199B43, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:57:41,756 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:41,756 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E67DAADF713241B9AE670519629E424C: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:41,758 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:41,760 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:41,762 INFO output.MROutput (MROutput.java:close(526)) - C762E2AC9A5D4070ABC00C5837E758E4 closed
2016-05-18 00:57:41,763 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533061479_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:41,763 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533061479_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:41,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533061479_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=20, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=16, LAST_EVENT_RECEIVED=16][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533061755, Process_Duration=8, Process_End_Time=1463533061763, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:41,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:41,764 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D00A46C5316A4874B0AE33E0BE199B43, mem on close (mb), free: 295, total: 362, max: 672
2016-05-18 00:57:41,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=D00A46C5316A4874B0AE33E0BE199B43, index=1, interruptedStatus=false
2016-05-18 00:57:41,765 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E67DAADF713241B9AE670519629E424C
2016-05-18 00:57:41,765 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E67DAADF713241B9AE670519629E424C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:41,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=D00A46C5316A4874B0AE33E0BE199B43, sourceVertex=E67DAADF713241B9AE670519629E424C, interruptedStatus=true
2016-05-18 00:57:41,765 INFO output.MROutput (MROutput.java:close(526)) - C762E2AC9A5D4070ABC00C5837E758E4 closed
2016-05-18 00:57:41,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=D00A46C5316A4874B0AE33E0BE199B43, sourceVertex=C762E2AC9A5D4070ABC00C5837E758E4, interruptedStatus=true
2016-05-18 00:57:41,766 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533061479_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:41,766 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:41,766 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533061479_0001_00_000002
2016-05-18 00:57:41,766 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533061479_0001_00_000002
2016-05-18 00:57:41,766 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D00A46C5316A4874B0AE33E0BE199B43, taskAttemptId=attempt_1463533061479_0001_1_01_000000_0, creationTime=1463533061710, allocationTime=1463533061710, startTime=1463533061714, finishTime=1463533061766, timeTaken=52, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533061479_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=20, MERGE_PHASE_TIME=23, FIRST_EVENT_RECEIVED=16, LAST_EVENT_RECEIVED=16, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533061755, Process_Duration=8, Process_End_Time=1463533061763, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:41,767 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533061479_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:41,767 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533061479_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:41,767 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533061479_0001_00_000002 completed but with errors
2016-05-18 00:57:41,767 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:TASK_FINISHED]: vertexName=D00A46C5316A4874B0AE33E0BE199B43, taskId=task_1463533061479_0001_1_01_000000, startTime=1463533061714, finishTime=1463533061767, timeTaken=53, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533061479_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:41,767 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533061479_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:41,767 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533061479_0001_00_000002, stoppedTime=1463533061767, exitStatus=1
2016-05-18 00:57:41,768 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] because task failed: task_1463533061479_0001_1_01_000000
2016-05-18 00:57:41,768 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:41,768 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:41,768 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:41,768 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,768 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=C762E2AC9A5D4070ABC00C5837E758E4, vertexId=vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,769 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:VERTEX_FINISHED]: vertexName=D00A46C5316A4874B0AE33E0BE199B43, vertexId=vertex_1463533061479_0001_1_01, initRequestedTime=1463533061638, initedTime=1463533061644, startRequestedTime=1463533061652, startedTime=1463533061652, finishTime=1463533061769, timeTaken=117, status=FAILED, diagnostics=Task failed, taskId=task_1463533061479_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533061479_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533061714, firstTasksToStart=[ task_1463533061479_0001_1_01_000000 ], lastTaskFinishTime=1463533061766, lastTasksToFinish=[ task_1463533061479_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:41,769 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:41,769 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533061479_0001_1_01_000000_0 failed: true container: container_1463533061479_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:41,770 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:41,770 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:41,770 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:41,771 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533061479_0001_1, queueSize=0
2016-05-18 00:57:41,771 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533061479_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533061479_0001_1, startTime=1463533061636, finishTime=1463533061771, timeTaken=135, status=FAILED, diagnostics=Vertex failed, vertexName=D00A46C5316A4874B0AE33E0BE199B43, vertexId=vertex_1463533061479_0001_1_01, diagnostics=[Task failed, taskId=task_1463533061479_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533061479_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=250, AM_GC_TIME_MILLIS=12, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533061693, Process_Duration=2, Process_End_Time=1463533061695, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:41,771 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533061479_0001_1_01 [D00A46C5316A4874B0AE33E0BE199B43]
2016-05-18 00:57:41,771 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533061479_0001_1 finished with state: FAILED
2016-05-18 00:57:41,771 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533061479_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:41 Completed Dag: dag_1463533061479_0001_1
2016-05-18 00:57:41,772 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533061479_0001_1, dagState=FAILED
2016-05-18 00:57:41,772 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533061479_0001_1_post
2016-05-18 00:57:41,772 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:41,772 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:41,772 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:41,772 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2] every/(1/1) ...emblyplatform/0_VALUES_-1, with id=dag_1463533061479_0001_1
2016-05-18 00:57:41,772 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2] every/(1/1) ...emblyplatform/0_VALUES_-1, with id=dag_1463533061479_0001_1
2016-05-18 00:57:41,776 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...emblyplatform/0_VALUES_-1, applicationId=application_1463533061479_0001
2016-05-18 00:57:41,778 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:41,778 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:41,778 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:41,778 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:41,778 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:41,778 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:41,779 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:41,779 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:41,779 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:41,779 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:41,780 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:41,780 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:41,780 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:41,781 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:41,785 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E45B4558FCD24BEFA9C58A020AE83250/ED4AE74EC10B4D41AA29BB10648322B2/.tez/application_1463533061479_0001
2016-05-18 00:57:41,786 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:41,787 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...emblyplatform/0_VALUES_-1
2016-05-18 00:57:41,787 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:41,787 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_-1/_temporary
2016-05-18 00:57:41,788 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.326
2016-05-18 00:57:41,788 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_VALUES_-1
2016-05-18 00:57:41,811 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:41,811 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:41,836 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.002
2016-05-18 00:57:41,840 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.006
2016-05-18 00:57:41,840 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:41,840 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:41,840 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_VALUES_'field'
2016-05-18 00:57:41,874 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:41,874 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:41,917 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 00:57:41,930 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.020
2016-05-18 00:57:41,931 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:41,931 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:41,931 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:41,952 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:41,959 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_RESULTS
2016-05-18 00:57:42,026 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:42,026 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:42,027 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_RESULTS"]
2016-05-18 00:57:42,027 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:42,027 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:42,027 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:42,029 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...platform/0_VALUES_RESULTS
2016-05-18 00:57:42,029 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:42,047 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:42,052 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:42,052 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:42,052 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:42,063 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001 doesn't exist and is created
2016-05-18 00:57:42,080 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:42,080 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:42,081 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001_wd
2016-05-18 00:57:42,092 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533062052_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:42,093 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:42,093 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:42,095 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:42,095 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:42,095 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:42,095 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:42,095 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:42,097 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:42,097 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:42,097 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:42,098 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:42,098 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:42,098 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:42,098 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:42,098 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:42,099 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:42,099 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:42,099 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:42,099 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:42,101 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:42,102 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:42,102 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:42,102 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001_wd/localmode-log-dir
2016-05-18 00:57:42,102 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001_wd/localmode-log-dir/history.txt.appattempt_1463533062052_0001_000000, maxErrors=10
2016-05-18 00:57:42,102 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:42,103 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533062052_0001_000000, appSubmitTime=1463533062092, launchTime=1463533062092
2016-05-18 00:57:42,110 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:41450
2016-05-18 00:57:42,110 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:42,110 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533062052
2016-05-18 00:57:42,113 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533062052_0001_000000, startTime=1463533062113
2016-05-18 00:57:42,113 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:42,181 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:42,181 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:42,181 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...platform/0_VALUES_RESULTS, applicationId=application_1463533062052_0001, dagName=[969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97] every/(1/1) ...platform/0_VALUES_RESULTS
2016-05-18 00:57:42,197 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97] every/(1/1) ...platform/0_VALUES_RESULTS
2016-05-18 00:57:42,197 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:42,198 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533062052_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001_wd/localmode-log-dir/dag_1463533062052_0001_1.dot
2016-05-18 00:57:42,198 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001_wd/localmode-log-dir/dag_1463533062052_0001_1-tez-dag.pb.txt
2016-05-18 00:57:42,204 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533062052_0001_1
2016-05-18 00:57:42,205 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,205 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97] every/(1/1) ...platform/0_VALUES_RESULTS
2016-05-18 00:57:42 Running Dag: dag_1463533062052_0001_1
2016-05-18 00:57:42,213 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533062052_0001_1, submitTime=1463533062197
2016-05-18 00:57:42,214 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001/tez-conf.pb" } size: 36913 timestamp: 1463533062000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:42,214 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,214 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:42,214 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001/tez-conf.pb" } size: 36913 timestamp: 1463533062000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:42,214 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,215 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:42,215 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:42,215 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533062052_0001_1, initTime=1463533062215
2016-05-18 00:57:42,215 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062052_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:42,217 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:42,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:DAG_STARTED]: dagID=dag_1463533062052_0001_1, startTime=1463533062217
2016-05-18 00:57:42,217 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...platform/0_VALUES_RESULTS, applicationId=application_1463533062052_0001, dagName=[969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97] every/(1/1) ...platform/0_VALUES_RESULTS
2016-05-18 00:57:42,217 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062052_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:42,218 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F4581C6F7E144C23BE816CC997E2BCA0 : {3D0C191E63CC4D0BA6BB58BC6847BEBD={InputName=3D0C191E63CC4D0BA6BB58BC6847BEBD}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:42,218 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 3D0C191E63CC4D0BA6BB58BC6847BEBD, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:42,218 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,218 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,218 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,218 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,218 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:42,219 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,221 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 3D0C191E63CC4D0BA6BB58BC6847BEBD on vertex vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,222 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:42,222 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,222 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,222 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533062052_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:42,222 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94], numAdditionalOutputs=1
2016-05-18 00:57:42,223 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=3AB6155D0ED749A8BDA8079A21A557CE, vertex=vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:42,224 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 3D0C191E63CC4D0BA6BB58BC6847BEBD asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:42,225 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for BE297618D8BD44BEBD84248CFF85BD94:3AB6155D0ED749A8BDA8079A21A557CE using oldmapred API
2016-05-18 00:57:42,225 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=3AB6155D0ED749A8BDA8079A21A557CE, vertexName=BE297618D8BD44BEBD84248CFF85BD94, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:42,226 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BE297618D8BD44BEBD84248CFF85BD94, vertexId=vertex_1463533062052_0001_1_01, initRequestedTime=1463533062219, initedTime=1463533062222, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:42,226 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:42,227 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:42,227 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:42,227 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:42,228 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:42,228 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 3D0C191E63CC4D0BA6BB58BC6847BEBD on vertex vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,229 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:42,229 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {3D0C191E63CC4D0BA6BB58BC6847BEBD=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:42,229 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] parallelism set to 1
2016-05-18 00:57:42,229 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:42,230 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F4581C6F7E144C23BE816CC997E2BCA0, vertexId=vertex_1463533062052_0001_1_00, initRequestedTime=1463533062218, initedTime=1463533062230, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:42,230 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] with distanceFromRoot: 0
2016-05-18 00:57:42,230 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:42,230 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533062052_0001_1_00, startRequestedTime=1463533062219, startedTime=1463533062230
2016-05-18 00:57:42,230 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533062052_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:42,230 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:42,231 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533062052_0001_1_00 for vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] numStartedSources: 1 numSources: 1
2016-05-18 00:57:42,231 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] with distanceFromRoot: 1
2016-05-18 00:57:42,231 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533062052_0001_1_01, startRequestedTime=1463533062231, startedTime=1463533062231
2016-05-18 00:57:42,231 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:42,232 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F4581C6F7E144C23BE816CC997E2BCA0
2016-05-18 00:57:42,233 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,232 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: BE297618D8BD44BEBD84248CFF85BD94 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:42,233 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:TASK_STARTED]: vertexName=F4581C6F7E144C23BE816CC997E2BCA0, taskId=task_1463533062052_0001_1_00_000000, scheduledTime=1463533062233, launchTime=1463533062233
2016-05-18 00:57:42,234 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,234 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F4581C6F7E144C23BE816CC997E2BCA0 in vertex: BE297618D8BD44BEBD84248CFF85BD94 numBipartiteSourceTasks: 1
2016-05-18 00:57:42,234 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:42,235 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533062052_0001, with dag id: dag_1463533062052_0001_1
2016-05-18 00:57:42,236 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533062052_0001_00_000001, launchTime=1463533062236
2016-05-18 00:57:42,237 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062052_0001_00_000001
2016-05-18 00:57:42,238 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533062052_0001_1_00_000000_0] started. Is using containerId: [container_1463533062052_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:42,238 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F4581C6F7E144C23BE816CC997E2BCA0, taskAttemptId=attempt_1463533062052_0001_1_00_000000_0, startTime=1463533062238, containerId=container_1463533062052_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:42,238 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062052_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533062052_0001_1_00_000000_0
2016-05-18 00:57:42 Starting to run new task attempt: attempt_1463533062052_0001_1_00_000000_0
2016-05-18 00:57:42,239 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533062052_0001_1_00_000000_0
2016-05-18 00:57:42,239 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,240 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:42,240 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97] every/(1/1) ...platform/0_VALUES_RESULTS, VertexName: F4581C6F7E144C23BE816CC997E2BCA0, VertexParallelism: 1, TaskAttemptID:attempt_1463533062052_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=3D0C191E63CC4D0BA6BB58BC6847BEBD, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=BE297618D8BD44BEBD84248CFF85BD94, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:42,240 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:42,240 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533062052_0001_1_00_000000_0
2016-05-18 00:57:42,241 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533062052_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:42,241 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533062052_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:42,243 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:42,245 INFO input.MRInput (MRInput.java:initialize(440)) - 3D0C191E63CC4D0BA6BB58BC6847BEBD using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:42,245 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 3D0C191E63CC4D0BA6BB58BC6847BEBD
2016-05-18 00:57:42,246 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F4581C6F7E144C23BE816CC997E2BCA0, ordinal: 0
2016-05-18 00:57:42,247 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F4581C6F7E144C23BE816CC997E2BCA0, mem on start (mb), free: 186, total: 362, max: 672
2016-05-18 00:57:42,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:42,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:42,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:42,248 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:42,248 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[3D0C191E63CC4D0BA6BB58BC6847BEBD:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [BE297618D8BD44BEBD84248CFF85BD94:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:42,249 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:42,249 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:42,250 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:42,250 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[3D0C191E63CC4D0BA6BB58BC6847BEBD:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [BE297618D8BD44BEBD84248CFF85BD94:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:42,250 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:42,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 3D0C191E63CC4D0BA6BB58BC6847BEBD being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:42,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:42,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:42,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 3D0C191E63CC4D0BA6BB58BC6847BEBD
2016-05-18 00:57:42,252 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:42,252 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533062052_0001_1_00_000000_0
2016-05-18 00:57:42,254 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:42,254 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 3D0C191E63CC4D0BA6BB58BC6847BEBD initialized RecordReader from event
2016-05-18 00:57:42,255 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:42,255 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 3D0C191E63CC4D0BA6BB58BC6847BEBD
2016-05-18 00:57:42,256 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: AC07388F4776453686F6E70A1A32DA1E
2016-05-18 00:57:42,256 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] AC07388F4776453686F6E70A1A32DA1E
2016-05-18 00:57:42,257 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:42,259 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:42,259 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - BE297618D8BD44BEBD84248CFF85BD94 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@3f9b6bc7, comparator=cascading.tuple.hadoop.util.TupleComparator@74c0f670, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:42,259 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for BE297618D8BD44BEBD84248CFF85BD94: , UsingHashComparator=false
2016-05-18 00:57:42,272 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:42,272 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:42,286 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:42,286 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:42,286 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for BE297618D8BD44BEBD84248CFF85BD94: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:42,286 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - BE297618D8BD44BEBD84248CFF85BD94: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:42,286 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:42,287 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F4581C6F7E144C23BE816CC997E2BCA0, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:42,290 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533062052_0001_1_00_000000_0
2016-05-18 00:57:42,290 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F4581C6F7E144C23BE816CC997E2BCA0, mem on close (mb), free: 197, total: 363, max: 672
2016-05-18 00:57:42,290 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - BE297618D8BD44BEBD84248CFF85BD94: Starting flush of map output
2016-05-18 00:57:42,290 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - BE297618D8BD44BEBD84248CFF85BD94: Span0.length = 20, perItem = 6
2016-05-18 00:57:42,290 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - BE297618D8BD44BEBD84248CFF85BD94: done sorting span=0, length=20, time=0
2016-05-18 00:57:42,290 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - BE297618D8BD44BEBD84248CFF85BD94: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:42,292 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - BE297618D8BD44BEBD84248CFF85BD94: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001_wd/localmode-local-dir/output/attempt_1463533062052_0001_1_00_000000_0_10182_0/file.out
2016-05-18 00:57:42,295 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:42 Completed running task attempt: attempt_1463533062052_0001_1_00_000000_0
2016-05-18 00:57:42,296 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533062052_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:42,296 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533062052_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:42,296 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533062052_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533062287, Process_Duration=2, Process_End_Time=1463533062289, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:42,297 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533062052_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:42,297 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:42,297 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:42,297 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,297 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062052_0001_00_000001
2016-05-18 00:57:42,297 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F4581C6F7E144C23BE816CC997E2BCA0, taskAttemptId=attempt_1463533062052_0001_1_00_000000_0, creationTime=1463533062233, allocationTime=1463533062235, startTime=1463533062238, finishTime=1463533062297, timeTaken=59, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:42,298 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533062052_0001_1_00_000000_0
2016-05-18 00:57:42,298 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:TASK_FINISHED]: vertexName=F4581C6F7E144C23BE816CC997E2BCA0, taskId=task_1463533062052_0001_1_00_000000, startTime=1463533062238, finishTime=1463533062298, timeTaken=60, status=SUCCEEDED, successfulAttemptID=attempt_1463533062052_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533062287, Process_Duration=2, Process_End_Time=1463533062289, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:42,299 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:42,299 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0]
2016-05-18 00:57:42,299 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:VERTEX_FINISHED]: vertexName=F4581C6F7E144C23BE816CC997E2BCA0, vertexId=vertex_1463533062052_0001_1_00, initRequestedTime=1463533062218, initedTime=1463533062230, startRequestedTime=1463533062219, startedTime=1463533062230, finishTime=1463533062299, timeTaken=69, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533062287, Process_Duration=2, Process_End_Time=1463533062289, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533062238, firstTasksToStart=[ task_1463533062052_0001_1_00_000000 ], lastTaskFinishTime=1463533062297, lastTasksToFinish=[ task_1463533062052_0001_1_00_000000 ], minTaskDuration=59, maxTaskDuration=59, avgTaskDuration=59.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533062052_0001_1_00_000000 ], longestDurationTasks=[ task_1463533062052_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:42,299 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:42,300 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533062052_0001_00_000001, stoppedTime=1463533062300, exitStatus=0
2016-05-18 00:57:42,300 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] attempt: attempt_1463533062052_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:42,300 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533062052_0001_1_00 [F4581C6F7E144C23BE816CC997E2BCA0] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:42,300 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:42,300 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533062052_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:42,301 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: BE297618D8BD44BEBD84248CFF85BD94
2016-05-18 00:57:42,301 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F4581C6F7E144C23BE816CC997E2BCA0, destinationVertexName=BE297618D8BD44BEBD84248CFF85BD94. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:42,301 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,301 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:TASK_STARTED]: vertexName=BE297618D8BD44BEBD84248CFF85BD94, taskId=task_1463533062052_0001_1_01_000000, scheduledTime=1463533062301, launchTime=1463533062301
2016-05-18 00:57:42,301 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,303 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533062052_0001_00_000002, launchTime=1463533062303
2016-05-18 00:57:42,303 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062052_0001_00_000002
2016-05-18 00:57:42,304 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062052_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533062052_0001_1_01_000000_0
2016-05-18 00:57:42,304 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533062052_0001_1_01_000000_0] started. Is using containerId: [container_1463533062052_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:42,304 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BE297618D8BD44BEBD84248CFF85BD94, taskAttemptId=attempt_1463533062052_0001_1_01_000000_0, startTime=1463533062304, containerId=container_1463533062052_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:42 Starting to run new task attempt: attempt_1463533062052_0001_1_01_000000_0
2016-05-18 00:57:42,305 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533062052_0001_1_01_000000_0
2016-05-18 00:57:42,305 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,305 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:42,305 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97] every/(1/1) ...platform/0_VALUES_RESULTS, VertexName: BE297618D8BD44BEBD84248CFF85BD94, VertexParallelism: 1, TaskAttemptID:attempt_1463533062052_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F4581C6F7E144C23BE816CC997E2BCA0, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=3AB6155D0ED749A8BDA8079A21A557CE, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:42,306 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:42,306 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533062052_0001_1_01_000000_0
2016-05-18 00:57:42,307 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533062052_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,307 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533062052_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:42,308 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533062052_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:42,308 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062052_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:42,308 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533062052_0001_00_000001, Exiting
2016-05-18 00:57:42,308 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533062052_0001_00_000001
2016-05-18 00:57:42,308 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533062052_0001_00_000001
2016-05-18 00:57:42,308 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:42,312 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:42,317 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BE297618D8BD44BEBD84248CFF85BD94, ordinal: 1
2016-05-18 00:57:42,317 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BE297618D8BD44BEBD84248CFF85BD94, mem on start (mb), free: 192, total: 363, max: 672
2016-05-18 00:57:42,317 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:42,317 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:42,317 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:42,319 INFO output.MROutput (MROutput.java:initialize(421)) - 3AB6155D0ED749A8BDA8079A21A557CE: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:42,319 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:42,319 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F4581C6F7E144C23BE816CC997E2BCA0:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [3AB6155D0ED749A8BDA8079A21A557CE:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:42,319 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533062052_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:42,319 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:42,319 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533062052_0001_00_000001 completed successfully
2016-05-18 00:57:42,319 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:42,320 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:42,320 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F4581C6F7E144C23BE816CC997E2BCA0:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [3AB6155D0ED749A8BDA8079A21A557CE:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:42,320 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:42,320 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F4581C6F7E144C23BE816CC997E2BCA0 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:42,320 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F4581C6F7E144C23BE816CC997E2BCA0: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:42,320 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F4581C6F7E144C23BE816CC997E2BCA0: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:42,320 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:42,320 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:42,321 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F4581C6F7E144C23BE816CC997E2BCA0 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:42,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F4581C6F7E144C23BE816CC997E2BCA0
2016-05-18 00:57:42,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:42,324 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533062052_0001_1_01_000000_0
2016-05-18 00:57:42,327 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:42,327 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: AC07388F4776453686F6E70A1A32DA1E
2016-05-18 00:57:42,327 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_RESULTS"], id: 3AB6155D0ED749A8BDA8079A21A557CE
2016-05-18 00:57:42,327 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F4581C6F7E144C23BE816CC997E2BCA0: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:42,327 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_RESULTS"]
2016-05-18 00:57:42,328 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] AC07388F4776453686F6E70A1A32DA1E, for 1 inputs
2016-05-18 00:57:42,330 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533062052_0001_1_00_000000_0_10182} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533062330, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:42,330 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F4581C6F7E144C23BE816CC997E2BCA0
2016-05-18 00:57:42,330 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:42,330 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F4581C6F7E144C23BE816CC997E2BCA0, wasInterrupted=false
2016-05-18 00:57:42,331 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:42,331 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:42,333 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:42,333 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001_wd/localmode-local-dir/output/attempt_1463533062052_0001_1_00_000000_0_10182/file.out, len=181, isLocal=true
2016-05-18 00:57:42,333 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:42,333 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:42,334 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:42,334 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:42,335 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F4581C6F7E144C23BE816CC997E2BCA0
2016-05-18 00:57:42,335 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BE297618D8BD44BEBD84248CFF85BD94, all 1 inputs ready in: 00:00:00.007
2016-05-18 00:57:42,335 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F4581C6F7E144C23BE816CC997E2BCA0: Shuffle Runner thread complete
2016-05-18 00:57:42,336 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:42,336 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F4581C6F7E144C23BE816CC997E2BCA0: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:42,338 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:42,339 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:42,339 INFO output.MROutput (MROutput.java:close(526)) - 3AB6155D0ED749A8BDA8079A21A557CE closed
2016-05-18 00:57:42,339 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533062052_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:42,340 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533062052_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:42,340 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533062052_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533062335, Process_Duration=5, Process_End_Time=1463533062340, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:42,340 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:42,340 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BE297618D8BD44BEBD84248CFF85BD94, mem on close (mb), free: 188, total: 363, max: 672
2016-05-18 00:57:42,341 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=BE297618D8BD44BEBD84248CFF85BD94, index=1, interruptedStatus=false
2016-05-18 00:57:42,341 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F4581C6F7E144C23BE816CC997E2BCA0
2016-05-18 00:57:42,341 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F4581C6F7E144C23BE816CC997E2BCA0: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:42,341 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=BE297618D8BD44BEBD84248CFF85BD94, sourceVertex=F4581C6F7E144C23BE816CC997E2BCA0, interruptedStatus=true
2016-05-18 00:57:42,341 INFO output.MROutput (MROutput.java:close(526)) - 3AB6155D0ED749A8BDA8079A21A557CE closed
2016-05-18 00:57:42,342 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=BE297618D8BD44BEBD84248CFF85BD94, sourceVertex=3AB6155D0ED749A8BDA8079A21A557CE, interruptedStatus=true
2016-05-18 00:57:42,342 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533062052_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:42,343 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:42,343 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533062052_0001_00_000002
2016-05-18 00:57:42,343 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BE297618D8BD44BEBD84248CFF85BD94, taskAttemptId=attempt_1463533062052_0001_1_01_000000_0, creationTime=1463533062301, allocationTime=1463533062302, startTime=1463533062304, finishTime=1463533062342, timeTaken=38, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533062052_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=11, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533062335, Process_Duration=5, Process_End_Time=1463533062340, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:42,343 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533062052_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:42,343 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:TASK_FINISHED]: vertexName=BE297618D8BD44BEBD84248CFF85BD94, taskId=task_1463533062052_0001_1_01_000000, startTime=1463533062304, finishTime=1463533062343, timeTaken=39, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062052_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:42,343 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] because task failed: task_1463533062052_0001_1_01_000000
2016-05-18 00:57:42,344 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:42,344 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:42,344 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:42,344 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,344 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=3AB6155D0ED749A8BDA8079A21A557CE, vertexId=vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,343 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533062052_0001_00_000002
2016-05-18 00:57:42,345 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533062052_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:42,345 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:VERTEX_FINISHED]: vertexName=BE297618D8BD44BEBD84248CFF85BD94, vertexId=vertex_1463533062052_0001_1_01, initRequestedTime=1463533062219, initedTime=1463533062222, startRequestedTime=1463533062231, startedTime=1463533062231, finishTime=1463533062345, timeTaken=114, status=FAILED, diagnostics=Task failed, taskId=task_1463533062052_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062052_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533062304, firstTasksToStart=[ task_1463533062052_0001_1_01_000000 ], lastTaskFinishTime=1463533062342, lastTasksToFinish=[ task_1463533062052_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:42,345 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:42,345 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533062052_0001_00_000002 completed but with errors
2016-05-18 00:57:42,346 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533062052_0001_00_000002, stoppedTime=1463533062346, exitStatus=-1000
2016-05-18 00:57:42,346 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533062052_0001_1_01_000000_0 failed: true container: container_1463533062052_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:42,347 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:42,347 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:42,347 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:42,348 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533062052_0001_1, queueSize=0
2016-05-18 00:57:42,348 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062052_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533062052_0001_1, startTime=1463533062217, finishTime=1463533062348, timeTaken=131, status=FAILED, diagnostics=Vertex failed, vertexName=BE297618D8BD44BEBD84248CFF85BD94, vertexId=vertex_1463533062052_0001_1_01, diagnostics=[Task failed, taskId=task_1463533062052_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062052_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=240, AM_GC_TIME_MILLIS=12, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533062287, Process_Duration=2, Process_End_Time=1463533062289, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:42,349 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533062052_0001_1_01 [BE297618D8BD44BEBD84248CFF85BD94]
2016-05-18 00:57:42,349 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533062052_0001_1 finished with state: FAILED
2016-05-18 00:57:42,349 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062052_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:42,349 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533062052_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:42,349 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533062052_0001_00_000002
2016-05-18 00:57:42 Completed Dag: dag_1463533062052_0001_1
2016-05-18 00:57:42,349 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533062052_0001_1, dagState=FAILED
2016-05-18 00:57:42,350 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533062052_0001_1_post
2016-05-18 00:57:42,350 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,350 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 2
2016-05-18 00:57:42,350 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:42,350 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97] every/(1/1) ...platform/0_VALUES_RESULTS, with id=dag_1463533062052_0001_1
2016-05-18 00:57:42,350 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97] every/(1/1) ...platform/0_VALUES_RESULTS, with id=dag_1463533062052_0001_1
2016-05-18 00:57:42,356 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...platform/0_VALUES_RESULTS, applicationId=application_1463533062052_0001
2016-05-18 00:57:42,358 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:42,358 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:42,358 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:42,358 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:42,358 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:42,358 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:42,359 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:42,359 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:42,359 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:42,361 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:42,362 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:42,362 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:42,362 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:42,363 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:42,366 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/969774407AAF4CABBC6C54CBD004646D/B1AE782AE62240C4A3AD892ED29EBA97/.tez/application_1463533062052_0001
2016-05-18 00:57:42,366 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:42,367 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...platform/0_VALUES_RESULTS
2016-05-18 00:57:42,367 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:42,367 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_RESULTS/_temporary
2016-05-18 00:57:42,368 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.341
2016-05-18 00:57:42,368 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_VALUES_RESULTS
2016-05-18 00:57:42,396 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:42,396 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:42,426 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:42,433 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:42,433 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:42,433 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:42,434 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:42,447 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:42,452 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_ALL
2016-05-18 00:57:42,497 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:42,497 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:42,497 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_ALL"]
2016-05-18 00:57:42,497 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:42,497 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:42,499 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:42,499 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...mblyplatform/0_VALUES_ALL
2016-05-18 00:57:42,499 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:42,513 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:42,517 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:42,517 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:42,517 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:42,526 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001 doesn't exist and is created
2016-05-18 00:57:42,541 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:42,541 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:42,542 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001_wd
2016-05-18 00:57:42,551 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533062517_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:42,551 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:42,551 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:42,552 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:42,552 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:42,553 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:42,553 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:42,553 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:42,554 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:42,554 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:42,554 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:42,555 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:42,555 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:42,555 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:42,555 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:42,555 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:42,555 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:42,555 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:42,556 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:42,556 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:42,557 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:42,557 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:42,558 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:42,558 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001_wd/localmode-log-dir
2016-05-18 00:57:42,558 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001_wd/localmode-log-dir/history.txt.appattempt_1463533062517_0001_000000, maxErrors=10
2016-05-18 00:57:42,558 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:42,558 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533062517_0001_000000, appSubmitTime=1463533062550, launchTime=1463533062551
2016-05-18 00:57:42,562 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:42,563 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:52596
2016-05-18 00:57:42,566 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533062517
2016-05-18 00:57:42,567 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533062517_0001_000000, startTime=1463533062567
2016-05-18 00:57:42,567 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:42,641 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:42,641 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:42,642 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...mblyplatform/0_VALUES_ALL, applicationId=application_1463533062517_0001, dagName=[CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E] every/(1/1) ...mblyplatform/0_VALUES_ALL
2016-05-18 00:57:42,657 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E] every/(1/1) ...mblyplatform/0_VALUES_ALL
2016-05-18 00:57:42,657 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:42,657 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533062517_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001_wd/localmode-log-dir/dag_1463533062517_0001_1.dot
2016-05-18 00:57:42,658 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001_wd/localmode-log-dir/dag_1463533062517_0001_1-tez-dag.pb.txt
2016-05-18 00:57:42,664 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533062517_0001_1
2016-05-18 00:57:42,664 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,664 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E] every/(1/1) ...mblyplatform/0_VALUES_ALL
2016-05-18 00:57:42 Running Dag: dag_1463533062517_0001_1
2016-05-18 00:57:42,673 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533062517_0001_1, submitTime=1463533062657
2016-05-18 00:57:42,674 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001/tez-conf.pb" } size: 36913 timestamp: 1463533062000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:42,674 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,674 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:42,674 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001/tez-conf.pb" } size: 36913 timestamp: 1463533062000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:42,675 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,675 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:42,675 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:42,675 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533062517_0001_1, initTime=1463533062675
2016-05-18 00:57:42,675 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062517_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:42,676 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:42,676 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:DAG_STARTED]: dagID=dag_1463533062517_0001_1, startTime=1463533062676
2016-05-18 00:57:42,676 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062517_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:42,676 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A287BE323FD34F13A8857CB45D114C2A : {84E687B309FF4EC5835C52075297F050={InputName=84E687B309FF4EC5835C52075297F050}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:42,676 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 84E687B309FF4EC5835C52075297F050, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:42,676 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,677 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,677 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,677 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,677 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:42,678 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,679 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...mblyplatform/0_VALUES_ALL, applicationId=application_1463533062517_0001, dagName=[CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E] every/(1/1) ...mblyplatform/0_VALUES_ALL
2016-05-18 00:57:42,679 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 84E687B309FF4EC5835C52075297F050 on vertex vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,681 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:42,681 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,681 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 84E687B309FF4EC5835C52075297F050 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:42,681 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,682 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533062517_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:42,682 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001], numAdditionalOutputs=1
2016-05-18 00:57:42,682 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=BB916EE1426C4B75B7D9DCF5F11B515B, vertex=vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:42,684 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A5C3786955DF42EC8E47EB3EE62B5001:BB916EE1426C4B75B7D9DCF5F11B515B using oldmapred API
2016-05-18 00:57:42,684 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=BB916EE1426C4B75B7D9DCF5F11B515B, vertexName=A5C3786955DF42EC8E47EB3EE62B5001, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:42,685 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A5C3786955DF42EC8E47EB3EE62B5001, vertexId=vertex_1463533062517_0001_1_01, initRequestedTime=1463533062678, initedTime=1463533062682, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:42,685 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:42,687 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:42,688 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:42,688 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:42,688 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:42,689 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 84E687B309FF4EC5835C52075297F050 on vertex vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,690 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:42,690 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {84E687B309FF4EC5835C52075297F050=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:42,692 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] parallelism set to 1
2016-05-18 00:57:42,692 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:42,692 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A287BE323FD34F13A8857CB45D114C2A, vertexId=vertex_1463533062517_0001_1_00, initRequestedTime=1463533062676, initedTime=1463533062692, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:42,692 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] with distanceFromRoot: 0
2016-05-18 00:57:42,692 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:42,692 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533062517_0001_1_00, startRequestedTime=1463533062678, startedTime=1463533062692
2016-05-18 00:57:42,692 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533062517_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:42,693 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:42,693 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533062517_0001_1_00 for vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] numStartedSources: 1 numSources: 1
2016-05-18 00:57:42,693 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] with distanceFromRoot: 1
2016-05-18 00:57:42,693 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A287BE323FD34F13A8857CB45D114C2A
2016-05-18 00:57:42,693 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,693 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533062517_0001_1_01, startRequestedTime=1463533062693, startedTime=1463533062693
2016-05-18 00:57:42,693 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:42,694 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:TASK_STARTED]: vertexName=A287BE323FD34F13A8857CB45D114C2A, taskId=task_1463533062517_0001_1_00_000000, scheduledTime=1463533062694, launchTime=1463533062694
2016-05-18 00:57:42,694 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,695 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A5C3786955DF42EC8E47EB3EE62B5001 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:42,695 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:42,697 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A287BE323FD34F13A8857CB45D114C2A in vertex: A5C3786955DF42EC8E47EB3EE62B5001 numBipartiteSourceTasks: 1
2016-05-18 00:57:42,699 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533062517_0001_00_000001, launchTime=1463533062698
2016-05-18 00:57:42,699 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062517_0001_00_000001
2016-05-18 00:57:42,700 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062517_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533062517_0001_1_00_000000_0
2016-05-18 00:57:42,700 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533062517_0001_1_00_000000_0] started. Is using containerId: [container_1463533062517_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:42,700 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A287BE323FD34F13A8857CB45D114C2A, taskAttemptId=attempt_1463533062517_0001_1_00_000000_0, startTime=1463533062700, containerId=container_1463533062517_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:42 Starting to run new task attempt: attempt_1463533062517_0001_1_00_000000_0
2016-05-18 00:57:42,700 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533062517_0001_1_00_000000_0
2016-05-18 00:57:42,700 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,701 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:42,701 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E] every/(1/1) ...mblyplatform/0_VALUES_ALL, VertexName: A287BE323FD34F13A8857CB45D114C2A, VertexParallelism: 1, TaskAttemptID:attempt_1463533062517_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=84E687B309FF4EC5835C52075297F050, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A5C3786955DF42EC8E47EB3EE62B5001, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:42,701 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:42,703 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533062517_0001_1_00_000000_0
2016-05-18 00:57:42,704 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533062517_0001, with dag id: dag_1463533062517_0001_1
2016-05-18 00:57:42,704 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533062517_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:42,704 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533062517_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:42,707 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:42,710 INFO input.MRInput (MRInput.java:initialize(440)) - 84E687B309FF4EC5835C52075297F050 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:42,711 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 84E687B309FF4EC5835C52075297F050
2016-05-18 00:57:42,713 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A287BE323FD34F13A8857CB45D114C2A, ordinal: 0
2016-05-18 00:57:42,713 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A287BE323FD34F13A8857CB45D114C2A, mem on start (mb), free: 103, total: 363, max: 672
2016-05-18 00:57:42,713 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:42,713 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:42,713 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:42,713 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:42,713 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[84E687B309FF4EC5835C52075297F050:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [A5C3786955DF42EC8E47EB3EE62B5001:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:42,713 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:42,713 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:42,714 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:42,714 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[84E687B309FF4EC5835C52075297F050:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [A5C3786955DF42EC8E47EB3EE62B5001:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:42,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:42,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 84E687B309FF4EC5835C52075297F050 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:42,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:42,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:42,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 84E687B309FF4EC5835C52075297F050
2016-05-18 00:57:42,714 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:42,715 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533062517_0001_1_00_000000_0
2016-05-18 00:57:42,716 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:42,716 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 84E687B309FF4EC5835C52075297F050 initialized RecordReader from event
2016-05-18 00:57:42,717 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:42,717 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 84E687B309FF4EC5835C52075297F050
2016-05-18 00:57:42,717 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 110B3DCCC4114C219C0444920620F8DF
2016-05-18 00:57:42,718 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 110B3DCCC4114C219C0444920620F8DF
2016-05-18 00:57:42,718 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:42,720 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:42,720 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A5C3786955DF42EC8E47EB3EE62B5001 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@75d6074e, comparator=cascading.tuple.hadoop.util.TupleComparator@3679ce52, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:42,721 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A5C3786955DF42EC8E47EB3EE62B5001: , UsingHashComparator=false
2016-05-18 00:57:42,749 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:42,749 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:42,749 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A5C3786955DF42EC8E47EB3EE62B5001: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:42,749 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A5C3786955DF42EC8E47EB3EE62B5001: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:42,749 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:42,749 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A287BE323FD34F13A8857CB45D114C2A, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:42,752 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533062517_0001_1_00_000000_0
2016-05-18 00:57:42,752 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A287BE323FD34F13A8857CB45D114C2A, mem on close (mb), free: 197, total: 364, max: 672
2016-05-18 00:57:42,752 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A5C3786955DF42EC8E47EB3EE62B5001: Starting flush of map output
2016-05-18 00:57:42,753 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A5C3786955DF42EC8E47EB3EE62B5001: Span0.length = 20, perItem = 6
2016-05-18 00:57:42,753 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A5C3786955DF42EC8E47EB3EE62B5001: done sorting span=0, length=20, time=0
2016-05-18 00:57:42,753 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A5C3786955DF42EC8E47EB3EE62B5001: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:42,754 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A5C3786955DF42EC8E47EB3EE62B5001: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001_wd/localmode-local-dir/output/attempt_1463533062517_0001_1_00_000000_0_10189_0/file.out
2016-05-18 00:57:42,758 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:42 Completed running task attempt: attempt_1463533062517_0001_1_00_000000_0
2016-05-18 00:57:42,758 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533062517_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:42,758 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533062517_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:42,759 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533062517_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533062749, Process_Duration=3, Process_End_Time=1463533062752, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:42,759 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533062517_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:42,759 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:42,759 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A287BE323FD34F13A8857CB45D114C2A, taskAttemptId=attempt_1463533062517_0001_1_00_000000_0, creationTime=1463533062694, allocationTime=1463533062698, startTime=1463533062700, finishTime=1463533062759, timeTaken=59, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:42,759 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:42,760 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,760 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533062517_0001_1_00_000000_0
2016-05-18 00:57:42,760 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062517_0001_00_000001
2016-05-18 00:57:42,760 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:TASK_FINISHED]: vertexName=A287BE323FD34F13A8857CB45D114C2A, taskId=task_1463533062517_0001_1_00_000000, startTime=1463533062700, finishTime=1463533062760, timeTaken=60, status=SUCCEEDED, successfulAttemptID=attempt_1463533062517_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533062749, Process_Duration=3, Process_End_Time=1463533062752, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:42,761 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:42,761 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A]
2016-05-18 00:57:42,761 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:VERTEX_FINISHED]: vertexName=A287BE323FD34F13A8857CB45D114C2A, vertexId=vertex_1463533062517_0001_1_00, initRequestedTime=1463533062676, initedTime=1463533062692, startRequestedTime=1463533062678, startedTime=1463533062692, finishTime=1463533062761, timeTaken=69, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533062749, Process_Duration=3, Process_End_Time=1463533062752, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533062700, firstTasksToStart=[ task_1463533062517_0001_1_00_000000 ], lastTaskFinishTime=1463533062759, lastTasksToFinish=[ task_1463533062517_0001_1_00_000000 ], minTaskDuration=59, maxTaskDuration=59, avgTaskDuration=59.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533062517_0001_1_00_000000 ], longestDurationTasks=[ task_1463533062517_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:42,762 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:42,762 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] attempt: attempt_1463533062517_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:42,762 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A5C3786955DF42EC8E47EB3EE62B5001
2016-05-18 00:57:42,762 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A287BE323FD34F13A8857CB45D114C2A, destinationVertexName=A5C3786955DF42EC8E47EB3EE62B5001. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:42,762 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,762 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533062517_0001_00_000001, stoppedTime=1463533062762, exitStatus=0
2016-05-18 00:57:42,762 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533062517_0001_1_00 [A287BE323FD34F13A8857CB45D114C2A] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:42,763 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:42,763 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:TASK_STARTED]: vertexName=A5C3786955DF42EC8E47EB3EE62B5001, taskId=task_1463533062517_0001_1_01_000000, scheduledTime=1463533062763, launchTime=1463533062763
2016-05-18 00:57:42,763 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,763 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533062517_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:42,765 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533062517_0001_00_000002, launchTime=1463533062765
2016-05-18 00:57:42,765 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062517_0001_00_000002
2016-05-18 00:57:42,766 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533062517_0001_1_01_000000_0] started. Is using containerId: [container_1463533062517_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:42,766 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062517_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533062517_0001_1_01_000000_0
2016-05-18 00:57:42,766 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A5C3786955DF42EC8E47EB3EE62B5001, taskAttemptId=attempt_1463533062517_0001_1_01_000000_0, startTime=1463533062766, containerId=container_1463533062517_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:42 Starting to run new task attempt: attempt_1463533062517_0001_1_01_000000_0
2016-05-18 00:57:42,767 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533062517_0001_1_01_000000_0
2016-05-18 00:57:42,767 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,767 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:42,767 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E] every/(1/1) ...mblyplatform/0_VALUES_ALL, VertexName: A5C3786955DF42EC8E47EB3EE62B5001, VertexParallelism: 1, TaskAttemptID:attempt_1463533062517_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A287BE323FD34F13A8857CB45D114C2A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=BB916EE1426C4B75B7D9DCF5F11B515B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:42,767 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:42,768 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533062517_0001_1_01_000000_0
2016-05-18 00:57:42,769 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533062517_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,769 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533062517_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:42,771 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533062517_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:42,771 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062517_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:42,772 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533062517_0001_00_000001, Exiting
2016-05-18 00:57:42,772 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533062517_0001_00_000001
2016-05-18 00:57:42,772 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533062517_0001_00_000001
2016-05-18 00:57:42,772 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:42,774 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533062517_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:42,774 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533062517_0001_00_000001 completed successfully
2016-05-18 00:57:42,775 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:42,777 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A5C3786955DF42EC8E47EB3EE62B5001, ordinal: 1
2016-05-18 00:57:42,777 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A5C3786955DF42EC8E47EB3EE62B5001, mem on start (mb), free: 191, total: 364, max: 672
2016-05-18 00:57:42,777 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:42,777 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:42,777 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:42,779 INFO output.MROutput (MROutput.java:initialize(421)) - BB916EE1426C4B75B7D9DCF5F11B515B: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:42,779 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:42,779 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BB916EE1426C4B75B7D9DCF5F11B515B:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A287BE323FD34F13A8857CB45D114C2A:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:42,779 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:42,780 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:42,780 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:42,780 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BB916EE1426C4B75B7D9DCF5F11B515B:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A287BE323FD34F13A8857CB45D114C2A:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:42,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:42,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A287BE323FD34F13A8857CB45D114C2A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:42,780 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A287BE323FD34F13A8857CB45D114C2A: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:42,781 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A287BE323FD34F13A8857CB45D114C2A: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:42,782 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A287BE323FD34F13A8857CB45D114C2A with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:42,782 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:42,782 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:42,784 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A287BE323FD34F13A8857CB45D114C2A
2016-05-18 00:57:42,784 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:42,784 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533062517_0001_1_01_000000_0
2016-05-18 00:57:42,784 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A287BE323FD34F13A8857CB45D114C2A: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:42,786 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533062517_0001_1_00_000000_0_10189} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533062786, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:42,786 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A287BE323FD34F13A8857CB45D114C2A
2016-05-18 00:57:42,788 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:42,788 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A287BE323FD34F13A8857CB45D114C2A, wasInterrupted=false
2016-05-18 00:57:42,788 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:42,789 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:42,790 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:42,790 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001_wd/localmode-local-dir/output/attempt_1463533062517_0001_1_00_000000_0_10189/file.out, len=181, isLocal=true
2016-05-18 00:57:42,790 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:42,790 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:42,790 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:42,790 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 110B3DCCC4114C219C0444920620F8DF
2016-05-18 00:57:42,790 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_ALL"], id: BB916EE1426C4B75B7D9DCF5F11B515B
2016-05-18 00:57:42,791 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_ALL"]
2016-05-18 00:57:42,791 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 110B3DCCC4114C219C0444920620F8DF, for 1 inputs
2016-05-18 00:57:42,790 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:42,791 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:42,791 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A287BE323FD34F13A8857CB45D114C2A
2016-05-18 00:57:42,791 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A287BE323FD34F13A8857CB45D114C2A: Shuffle Runner thread complete
2016-05-18 00:57:42,791 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A5C3786955DF42EC8E47EB3EE62B5001, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:42,793 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:42,793 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A287BE323FD34F13A8857CB45D114C2A: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:42,795 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:42,796 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:42,796 INFO output.MROutput (MROutput.java:close(526)) - BB916EE1426C4B75B7D9DCF5F11B515B closed
2016-05-18 00:57:42,796 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533062517_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:42,797 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533062517_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:42,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533062517_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533062792, Process_Duration=5, Process_End_Time=1463533062797, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:42,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:42,798 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A5C3786955DF42EC8E47EB3EE62B5001, mem on close (mb), free: 188, total: 364, max: 672
2016-05-18 00:57:42,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=A5C3786955DF42EC8E47EB3EE62B5001, index=1, interruptedStatus=false
2016-05-18 00:57:42,799 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A287BE323FD34F13A8857CB45D114C2A
2016-05-18 00:57:42,799 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A287BE323FD34F13A8857CB45D114C2A: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:42,799 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=A5C3786955DF42EC8E47EB3EE62B5001, sourceVertex=A287BE323FD34F13A8857CB45D114C2A, interruptedStatus=true
2016-05-18 00:57:42,799 INFO output.MROutput (MROutput.java:close(526)) - BB916EE1426C4B75B7D9DCF5F11B515B closed
2016-05-18 00:57:42,799 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=A5C3786955DF42EC8E47EB3EE62B5001, sourceVertex=BB916EE1426C4B75B7D9DCF5F11B515B, interruptedStatus=true
2016-05-18 00:57:42,800 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533062517_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:42,800 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:42,800 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533062517_0001_00_000002
2016-05-18 00:57:42,800 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A5C3786955DF42EC8E47EB3EE62B5001, taskAttemptId=attempt_1463533062517_0001_1_01_000000_0, creationTime=1463533062763, allocationTime=1463533062764, startTime=1463533062766, finishTime=1463533062800, timeTaken=34, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533062517_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533062792, Process_Duration=5, Process_End_Time=1463533062797, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:42,800 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533062517_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:42,800 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533062517_0001_00_000002
2016-05-18 00:57:42,801 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:TASK_FINISHED]: vertexName=A5C3786955DF42EC8E47EB3EE62B5001, taskId=task_1463533062517_0001_1_01_000000, startTime=1463533062766, finishTime=1463533062801, timeTaken=35, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062517_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:42,801 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533062517_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:42,801 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533062517_0001_00_000002 completed but with errors
2016-05-18 00:57:42,801 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] because task failed: task_1463533062517_0001_1_01_000000
2016-05-18 00:57:42,801 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:42,801 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:42,801 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:42,801 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,801 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=BB916EE1426C4B75B7D9DCF5F11B515B, vertexId=vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,803 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:VERTEX_FINISHED]: vertexName=A5C3786955DF42EC8E47EB3EE62B5001, vertexId=vertex_1463533062517_0001_1_01, initRequestedTime=1463533062678, initedTime=1463533062682, startRequestedTime=1463533062693, startedTime=1463533062693, finishTime=1463533062802, timeTaken=109, status=FAILED, diagnostics=Task failed, taskId=task_1463533062517_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062517_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533062766, firstTasksToStart=[ task_1463533062517_0001_1_01_000000 ], lastTaskFinishTime=1463533062800, lastTasksToFinish=[ task_1463533062517_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:42,803 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:42,803 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533062517_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:42,803 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533062517_0001_00_000002, stoppedTime=1463533062803, exitStatus=1
2016-05-18 00:57:42,803 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533062517_0001_1_01_000000_0 failed: true container: container_1463533062517_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:42,804 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:42,804 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:42,804 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:42,805 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533062517_0001_1, queueSize=0
2016-05-18 00:57:42,806 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062517_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533062517_0001_1, startTime=1463533062676, finishTime=1463533062805, timeTaken=129, status=FAILED, diagnostics=Vertex failed, vertexName=A5C3786955DF42EC8E47EB3EE62B5001, vertexId=vertex_1463533062517_0001_1_01, diagnostics=[Task failed, taskId=task_1463533062517_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062517_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=250, AM_GC_TIME_MILLIS=13, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533062749, Process_Duration=3, Process_End_Time=1463533062752, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:42,806 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533062517_0001_1_01 [A5C3786955DF42EC8E47EB3EE62B5001]
2016-05-18 00:57:42,806 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533062517_0001_1 finished with state: FAILED
2016-05-18 00:57:42,807 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062517_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:42 Completed Dag: dag_1463533062517_0001_1
2016-05-18 00:57:42,807 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533062517_0001_1, dagState=FAILED
2016-05-18 00:57:42,807 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533062517_0001_1_post
2016-05-18 00:57:42,807 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:42,807 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:42,808 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:42,808 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E] every/(1/1) ...mblyplatform/0_VALUES_ALL, with id=dag_1463533062517_0001_1
2016-05-18 00:57:42,808 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E] every/(1/1) ...mblyplatform/0_VALUES_ALL, with id=dag_1463533062517_0001_1
2016-05-18 00:57:42,813 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...mblyplatform/0_VALUES_ALL, applicationId=application_1463533062517_0001
2016-05-18 00:57:42,815 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:42,815 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:42,815 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:42,815 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:42,815 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:42,815 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:42,815 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:42,815 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:42,815 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:42,816 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:42,816 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:42,816 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:42,816 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:42,817 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:42,820 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/CE18BECAE59A491EB010969F6EDF2954/F5781792C1054522A21082666C250E5E/.tez/application_1463533062517_0001
2016-05-18 00:57:42,821 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:42,821 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...mblyplatform/0_VALUES_ALL
2016-05-18 00:57:42,822 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:42,822 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_ALL/_temporary
2016-05-18 00:57:42,822 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.325
2016-05-18 00:57:42,822 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_VALUES_ALL
2016-05-18 00:57:42,828 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:42,828 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:42,851 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:42,851 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:42,882 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:42,887 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 00:57:42,888 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:42,888 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:42,888 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:42,901 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:42,906 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_REPLACE
2016-05-18 00:57:42,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:42,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:42,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_REPLACE"]
2016-05-18 00:57:42,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:42,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:42,950 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:42,951 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...platform/0_VALUES_REPLACE
2016-05-18 00:57:42,951 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:42,963 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:42,967 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:42,967 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:42,967 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:42,976 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001 doesn't exist and is created
2016-05-18 00:57:42,990 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:42,990 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:42,991 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001_wd
2016-05-18 00:57:43,000 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533062967_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:43,000 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:43,000 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:43,002 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:43,002 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:43,002 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:43,002 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:43,004 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:43,005 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:43,005 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:43,005 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:43,006 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:43,006 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:43,006 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:43,006 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:43,006 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:43,007 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:43,008 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:43,008 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:43,008 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:43,010 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:43,010 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:43,010 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:43,011 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001_wd/localmode-log-dir
2016-05-18 00:57:43,011 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001_wd/localmode-log-dir/history.txt.appattempt_1463533062967_0001_000000, maxErrors=10
2016-05-18 00:57:43,012 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:43,012 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533062967_0001_000000, appSubmitTime=1463533062999, launchTime=1463533062999
2016-05-18 00:57:43,016 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:43,017 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:33063
2016-05-18 00:57:43,017 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533062967
2016-05-18 00:57:43,019 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533062967_0001_000000, startTime=1463533063019
2016-05-18 00:57:43,019 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:43,090 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:43,091 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:43,091 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...platform/0_VALUES_REPLACE, applicationId=application_1463533062967_0001, dagName=[2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8] every/(1/1) ...platform/0_VALUES_REPLACE
2016-05-18 00:57:43,113 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8] every/(1/1) ...platform/0_VALUES_REPLACE
2016-05-18 00:57:43,113 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:43,114 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533062967_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001_wd/localmode-log-dir/dag_1463533062967_0001_1.dot
2016-05-18 00:57:43,115 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001_wd/localmode-log-dir/dag_1463533062967_0001_1-tez-dag.pb.txt
2016-05-18 00:57:43,122 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533062967_0001_1
2016-05-18 00:57:43,122 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,122 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8] every/(1/1) ...platform/0_VALUES_REPLACE
2016-05-18 00:57:43 Running Dag: dag_1463533062967_0001_1
2016-05-18 00:57:43,130 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533062967_0001_1, submitTime=1463533063113
2016-05-18 00:57:43,131 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001/tez-conf.pb" } size: 36913 timestamp: 1463533062000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:43,131 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,131 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:43,132 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001/tez-conf.pb" } size: 36913 timestamp: 1463533062000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:43,133 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,133 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:43,133 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:43,133 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533062967_0001_1, initTime=1463533063133
2016-05-18 00:57:43,133 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062967_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:43,134 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:43,134 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:DAG_STARTED]: dagID=dag_1463533062967_0001_1, startTime=1463533063134
2016-05-18 00:57:43,134 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062967_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:43,134 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: BFE41DE1BDFD46A7A71A11F2895406DD : {82879810C36946BFAD445A4EE341C10C={InputName=82879810C36946BFAD445A4EE341C10C}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:43,134 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 82879810C36946BFAD445A4EE341C10C, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:43,134 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,134 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,134 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,134 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,135 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:43,135 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...platform/0_VALUES_REPLACE, applicationId=application_1463533062967_0001, dagName=[2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8] every/(1/1) ...platform/0_VALUES_REPLACE
2016-05-18 00:57:43,135 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,136 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 82879810C36946BFAD445A4EE341C10C on vertex vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,138 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 82879810C36946BFAD445A4EE341C10C asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:43,138 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:43,138 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,138 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,139 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533062967_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:43,139 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B], numAdditionalOutputs=1
2016-05-18 00:57:43,139 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=0A5E9F239CB940C8A887C7291333D2F6, vertex=vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:43,139 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:43,141 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:43,141 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A9EF8699F3604FFC9976B118650AB92B:0A5E9F239CB940C8A887C7291333D2F6 using oldmapred API
2016-05-18 00:57:43,141 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=0A5E9F239CB940C8A887C7291333D2F6, vertexName=A9EF8699F3604FFC9976B118650AB92B, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:43,141 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:43,142 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:43,142 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 82879810C36946BFAD445A4EE341C10C on vertex vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,143 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A9EF8699F3604FFC9976B118650AB92B, vertexId=vertex_1463533062967_0001_1_01, initRequestedTime=1463533063135, initedTime=1463533063139, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:43,143 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:43,143 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:43,144 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {82879810C36946BFAD445A4EE341C10C=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:43,146 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] parallelism set to 1
2016-05-18 00:57:43,146 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:43,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BFE41DE1BDFD46A7A71A11F2895406DD, vertexId=vertex_1463533062967_0001_1_00, initRequestedTime=1463533063134, initedTime=1463533063146, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:43,146 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] with distanceFromRoot: 0
2016-05-18 00:57:43,146 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:43,146 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533062967_0001_1_00, startRequestedTime=1463533063135, startedTime=1463533063146
2016-05-18 00:57:43,147 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533062967_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:43,147 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:43,147 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533062967_0001_1_00 for vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] numStartedSources: 1 numSources: 1
2016-05-18 00:57:43,147 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] with distanceFromRoot: 1
2016-05-18 00:57:43,147 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533062967_0001_1_01, startRequestedTime=1463533063147, startedTime=1463533063147
2016-05-18 00:57:43,147 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:43,148 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A9EF8699F3604FFC9976B118650AB92B with 0 source tasks and 1 pending tasks
2016-05-18 00:57:43,149 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in BFE41DE1BDFD46A7A71A11F2895406DD
2016-05-18 00:57:43,149 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BFE41DE1BDFD46A7A71A11F2895406DD in vertex: A9EF8699F3604FFC9976B118650AB92B numBipartiteSourceTasks: 1
2016-05-18 00:57:43,150 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,150 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:TASK_STARTED]: vertexName=BFE41DE1BDFD46A7A71A11F2895406DD, taskId=task_1463533062967_0001_1_00_000000, scheduledTime=1463533063150, launchTime=1463533063150
2016-05-18 00:57:43,150 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,151 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:43,153 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533062967_0001_00_000001, launchTime=1463533063152
2016-05-18 00:57:43,153 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062967_0001_00_000001
2016-05-18 00:57:43,154 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062967_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533062967_0001_1_00_000000_0
2016-05-18 00:57:43,154 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533062967_0001_1_00_000000_0] started. Is using containerId: [container_1463533062967_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:43,154 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BFE41DE1BDFD46A7A71A11F2895406DD, taskAttemptId=attempt_1463533062967_0001_1_00_000000_0, startTime=1463533063154, containerId=container_1463533062967_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:43 Starting to run new task attempt: attempt_1463533062967_0001_1_00_000000_0
2016-05-18 00:57:43,154 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533062967_0001_1_00_000000_0
2016-05-18 00:57:43,154 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,155 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:43,155 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8] every/(1/1) ...platform/0_VALUES_REPLACE, VertexName: BFE41DE1BDFD46A7A71A11F2895406DD, VertexParallelism: 1, TaskAttemptID:attempt_1463533062967_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=82879810C36946BFAD445A4EE341C10C, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A9EF8699F3604FFC9976B118650AB92B, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:43,155 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:43,155 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533062967_0001_1_00_000000_0
2016-05-18 00:57:43,156 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533062967_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:43,156 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533062967_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:43,157 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533062967_0001, with dag id: dag_1463533062967_0001_1
2016-05-18 00:57:43,160 INFO input.MRInput (MRInput.java:initialize(440)) - 82879810C36946BFAD445A4EE341C10C using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:43,160 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 82879810C36946BFAD445A4EE341C10C
2016-05-18 00:57:43,160 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:43,163 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BFE41DE1BDFD46A7A71A11F2895406DD, ordinal: 0
2016-05-18 00:57:43,163 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BFE41DE1BDFD46A7A71A11F2895406DD, mem on start (mb), free: 104, total: 364, max: 672
2016-05-18 00:57:43,163 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:43,163 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:43,163 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:43,163 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:43,163 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[82879810C36946BFAD445A4EE341C10C:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [A9EF8699F3604FFC9976B118650AB92B:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:43,164 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:43,164 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:43,164 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:43,164 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[82879810C36946BFAD445A4EE341C10C:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [A9EF8699F3604FFC9976B118650AB92B:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:43,164 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:43,164 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 82879810C36946BFAD445A4EE341C10C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:43,164 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:43,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:43,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 82879810C36946BFAD445A4EE341C10C
2016-05-18 00:57:43,167 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:43,167 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533062967_0001_1_00_000000_0
2016-05-18 00:57:43,169 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:43,170 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:43,170 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 82879810C36946BFAD445A4EE341C10C initialized RecordReader from event
2016-05-18 00:57:43,170 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 82879810C36946BFAD445A4EE341C10C
2016-05-18 00:57:43,170 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 6F379BF4221B467AB869D55FC08BA9E8
2016-05-18 00:57:43,170 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 6F379BF4221B467AB869D55FC08BA9E8
2016-05-18 00:57:43,171 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:43,174 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:43,174 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A9EF8699F3604FFC9976B118650AB92B using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@71ae3bf0, comparator=cascading.tuple.hadoop.util.TupleComparator@65e143de, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:43,174 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A9EF8699F3604FFC9976B118650AB92B: , UsingHashComparator=false
2016-05-18 00:57:43,198 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:43,199 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:43,199 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A9EF8699F3604FFC9976B118650AB92B: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:43,199 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A9EF8699F3604FFC9976B118650AB92B: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:43,200 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:43,200 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BFE41DE1BDFD46A7A71A11F2895406DD, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:43,204 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533062967_0001_1_00_000000_0
2016-05-18 00:57:43,204 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BFE41DE1BDFD46A7A71A11F2895406DD, mem on close (mb), free: 194, total: 363, max: 672
2016-05-18 00:57:43,204 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A9EF8699F3604FFC9976B118650AB92B: Starting flush of map output
2016-05-18 00:57:43,204 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A9EF8699F3604FFC9976B118650AB92B: Span0.length = 20, perItem = 6
2016-05-18 00:57:43,204 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A9EF8699F3604FFC9976B118650AB92B: done sorting span=0, length=20, time=0
2016-05-18 00:57:43,205 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A9EF8699F3604FFC9976B118650AB92B: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:43,206 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A9EF8699F3604FFC9976B118650AB92B: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001_wd/localmode-local-dir/output/attempt_1463533062967_0001_1_00_000000_0_10194_0/file.out
2016-05-18 00:57:43,209 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:43 Completed running task attempt: attempt_1463533062967_0001_1_00_000000_0
2016-05-18 00:57:43,209 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533062967_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:43,210 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533062967_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:43,210 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533062967_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533063200, Process_Duration=4, Process_End_Time=1463533063204, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:43,211 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533062967_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:43,211 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BFE41DE1BDFD46A7A71A11F2895406DD, taskAttemptId=attempt_1463533062967_0001_1_00_000000_0, creationTime=1463533063150, allocationTime=1463533063152, startTime=1463533063154, finishTime=1463533063211, timeTaken=57, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:43,211 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533062967_0001_1_00_000000_0
2016-05-18 00:57:43,211 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:43,212 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:43,212 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,212 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:TASK_FINISHED]: vertexName=BFE41DE1BDFD46A7A71A11F2895406DD, taskId=task_1463533062967_0001_1_00_000000, startTime=1463533063154, finishTime=1463533063211, timeTaken=57, status=SUCCEEDED, successfulAttemptID=attempt_1463533062967_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533063200, Process_Duration=4, Process_End_Time=1463533063204, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:43,212 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:43,213 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD]
2016-05-18 00:57:43,213 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062967_0001_00_000001
2016-05-18 00:57:43,213 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:VERTEX_FINISHED]: vertexName=BFE41DE1BDFD46A7A71A11F2895406DD, vertexId=vertex_1463533062967_0001_1_00, initRequestedTime=1463533063134, initedTime=1463533063146, startRequestedTime=1463533063135, startedTime=1463533063146, finishTime=1463533063213, timeTaken=67, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533063200, Process_Duration=4, Process_End_Time=1463533063204, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533063154, firstTasksToStart=[ task_1463533062967_0001_1_00_000000 ], lastTaskFinishTime=1463533063211, lastTasksToFinish=[ task_1463533062967_0001_1_00_000000 ], minTaskDuration=57, maxTaskDuration=57, avgTaskDuration=57.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533062967_0001_1_00_000000 ], longestDurationTasks=[ task_1463533062967_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:43,213 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:43,213 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533062967_0001_00_000001, stoppedTime=1463533063213, exitStatus=0
2016-05-18 00:57:43,214 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] attempt: attempt_1463533062967_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:43,214 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533062967_0001_1_00 [BFE41DE1BDFD46A7A71A11F2895406DD] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:43,214 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A9EF8699F3604FFC9976B118650AB92B
2016-05-18 00:57:43,214 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:43,214 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BFE41DE1BDFD46A7A71A11F2895406DD, destinationVertexName=A9EF8699F3604FFC9976B118650AB92B. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:43,214 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,214 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533062967_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:43,214 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:TASK_STARTED]: vertexName=A9EF8699F3604FFC9976B118650AB92B, taskId=task_1463533062967_0001_1_01_000000, scheduledTime=1463533063214, launchTime=1463533063214
2016-05-18 00:57:43,215 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533062967_0001_00_000002, launchTime=1463533063216
2016-05-18 00:57:43,217 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533062967_0001_00_000002
2016-05-18 00:57:43,217 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062967_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533062967_0001_1_01_000000_0
2016-05-18 00:57:43,218 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533062967_0001_1_01_000000_0] started. Is using containerId: [container_1463533062967_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:43,218 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A9EF8699F3604FFC9976B118650AB92B, taskAttemptId=attempt_1463533062967_0001_1_01_000000_0, startTime=1463533063217, containerId=container_1463533062967_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:43 Starting to run new task attempt: attempt_1463533062967_0001_1_01_000000_0
2016-05-18 00:57:43,219 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533062967_0001_1_01_000000_0
2016-05-18 00:57:43,219 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,219 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:43,220 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8] every/(1/1) ...platform/0_VALUES_REPLACE, VertexName: A9EF8699F3604FFC9976B118650AB92B, VertexParallelism: 1, TaskAttemptID:attempt_1463533062967_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BFE41DE1BDFD46A7A71A11F2895406DD, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=0A5E9F239CB940C8A887C7291333D2F6, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:43,220 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:43,220 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533062967_0001_1_01_000000_0
2016-05-18 00:57:43,221 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533062967_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,221 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533062967_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:43,223 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:43,223 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533062967_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:43,223 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533062967_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:43,223 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533062967_0001_00_000001, Exiting
2016-05-18 00:57:43,223 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533062967_0001_00_000001
2016-05-18 00:57:43,223 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533062967_0001_00_000001
2016-05-18 00:57:43,224 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:43,224 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533062967_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:43,224 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533062967_0001_00_000001 completed successfully
2016-05-18 00:57:43,228 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A9EF8699F3604FFC9976B118650AB92B, ordinal: 1
2016-05-18 00:57:43,228 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A9EF8699F3604FFC9976B118650AB92B, mem on start (mb), free: 189, total: 363, max: 672
2016-05-18 00:57:43,228 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:43,228 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:43,229 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:43,232 INFO output.MROutput (MROutput.java:initialize(421)) - 0A5E9F239CB940C8A887C7291333D2F6: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:43,232 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:43,232 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BFE41DE1BDFD46A7A71A11F2895406DD:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [0A5E9F239CB940C8A887C7291333D2F6:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:43,232 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:43,232 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:43,233 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:43,233 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BFE41DE1BDFD46A7A71A11F2895406DD:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [0A5E9F239CB940C8A887C7291333D2F6:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:43,233 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:43,233 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BFE41DE1BDFD46A7A71A11F2895406DD being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:43,233 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:43,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:43,234 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BFE41DE1BDFD46A7A71A11F2895406DD: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:43,234 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BFE41DE1BDFD46A7A71A11F2895406DD: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:43,235 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BFE41DE1BDFD46A7A71A11F2895406DD with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:43,236 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BFE41DE1BDFD46A7A71A11F2895406DD
2016-05-18 00:57:43,237 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:43,237 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533062967_0001_1_01_000000_0
2016-05-18 00:57:43,237 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BFE41DE1BDFD46A7A71A11F2895406DD: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:43,241 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533062967_0001_1_00_000000_0_10194} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533063241, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:43,242 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BFE41DE1BDFD46A7A71A11F2895406DD
2016-05-18 00:57:43,242 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:43,242 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BFE41DE1BDFD46A7A71A11F2895406DD, wasInterrupted=false
2016-05-18 00:57:43,243 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:43,243 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:43,243 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:43,244 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 6F379BF4221B467AB869D55FC08BA9E8
2016-05-18 00:57:43,244 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_REPLACE"], id: 0A5E9F239CB940C8A887C7291333D2F6
2016-05-18 00:57:43,244 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_REPLACE"]
2016-05-18 00:57:43,244 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 6F379BF4221B467AB869D55FC08BA9E8, for 1 inputs
2016-05-18 00:57:43,245 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:43,245 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001_wd/localmode-local-dir/output/attempt_1463533062967_0001_1_00_000000_0_10194/file.out, len=181, isLocal=true
2016-05-18 00:57:43,245 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:43,245 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:43,246 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:43,246 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:43,246 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BFE41DE1BDFD46A7A71A11F2895406DD
2016-05-18 00:57:43,246 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A9EF8699F3604FFC9976B118650AB92B, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:57:43,247 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BFE41DE1BDFD46A7A71A11F2895406DD: Shuffle Runner thread complete
2016-05-18 00:57:43,247 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:43,247 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BFE41DE1BDFD46A7A71A11F2895406DD: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:43,249 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:43,249 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:43,250 INFO output.MROutput (MROutput.java:close(526)) - 0A5E9F239CB940C8A887C7291333D2F6 closed
2016-05-18 00:57:43,250 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533062967_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:43,251 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533062967_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:43,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533062967_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533063246, Process_Duration=5, Process_End_Time=1463533063251, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:43,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:43,251 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A9EF8699F3604FFC9976B118650AB92B, mem on close (mb), free: 186, total: 363, max: 672
2016-05-18 00:57:43,251 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=A9EF8699F3604FFC9976B118650AB92B, index=1, interruptedStatus=false
2016-05-18 00:57:43,252 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BFE41DE1BDFD46A7A71A11F2895406DD
2016-05-18 00:57:43,253 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BFE41DE1BDFD46A7A71A11F2895406DD: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:43,253 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=A9EF8699F3604FFC9976B118650AB92B, sourceVertex=BFE41DE1BDFD46A7A71A11F2895406DD, interruptedStatus=true
2016-05-18 00:57:43,253 INFO output.MROutput (MROutput.java:close(526)) - 0A5E9F239CB940C8A887C7291333D2F6 closed
2016-05-18 00:57:43,254 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=A9EF8699F3604FFC9976B118650AB92B, sourceVertex=0A5E9F239CB940C8A887C7291333D2F6, interruptedStatus=true
2016-05-18 00:57:43,254 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533062967_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:43,254 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:43,255 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533062967_0001_00_000002
2016-05-18 00:57:43,255 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A9EF8699F3604FFC9976B118650AB92B, taskAttemptId=attempt_1463533062967_0001_1_01_000000_0, creationTime=1463533063215, allocationTime=1463533063216, startTime=1463533063217, finishTime=1463533063254, timeTaken=37, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533062967_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533063246, Process_Duration=5, Process_End_Time=1463533063251, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:43,255 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533062967_0001_00_000002
2016-05-18 00:57:43,255 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533062967_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:43,255 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:TASK_FINISHED]: vertexName=A9EF8699F3604FFC9976B118650AB92B, taskId=task_1463533062967_0001_1_01_000000, startTime=1463533063217, finishTime=1463533063255, timeTaken=38, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062967_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:43,255 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] because task failed: task_1463533062967_0001_1_01_000000
2016-05-18 00:57:43,255 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:43,256 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:43,256 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:43,256 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,255 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533062967_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:43,256 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=0A5E9F239CB940C8A887C7291333D2F6, vertexId=vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,256 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533062967_0001_00_000002 completed but with errors
2016-05-18 00:57:43,257 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:VERTEX_FINISHED]: vertexName=A9EF8699F3604FFC9976B118650AB92B, vertexId=vertex_1463533062967_0001_1_01, initRequestedTime=1463533063135, initedTime=1463533063139, startRequestedTime=1463533063147, startedTime=1463533063147, finishTime=1463533063257, timeTaken=110, status=FAILED, diagnostics=Task failed, taskId=task_1463533062967_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062967_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533063217, firstTasksToStart=[ task_1463533062967_0001_1_01_000000 ], lastTaskFinishTime=1463533063254, lastTasksToFinish=[ task_1463533062967_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:43,257 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:43,257 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533062967_0001_00_000002, stoppedTime=1463533063257, exitStatus=-1000
2016-05-18 00:57:43,257 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533062967_0001_1_01_000000_0 failed: true container: container_1463533062967_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:43,258 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533062967_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:43,258 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:43,258 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:43,258 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:43,259 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533062967_0001_1, queueSize=0
2016-05-18 00:57:43,259 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533062967_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533062967_0001_1, startTime=1463533063134, finishTime=1463533063259, timeTaken=125, status=FAILED, diagnostics=Vertex failed, vertexName=A9EF8699F3604FFC9976B118650AB92B, vertexId=vertex_1463533062967_0001_1_01, diagnostics=[Task failed, taskId=task_1463533062967_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533062967_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=220, AM_GC_TIME_MILLIS=11, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533063200, Process_Duration=4, Process_End_Time=1463533063204, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:43,259 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533062967_0001_1_01 [A9EF8699F3604FFC9976B118650AB92B]
2016-05-18 00:57:43,259 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533062967_0001_1 finished with state: FAILED
2016-05-18 00:57:43,259 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533062967_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:43,260 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533062967_0001_00_000002
2016-05-18 00:57:43 Completed Dag: dag_1463533062967_0001_1
2016-05-18 00:57:43,260 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533062967_0001_1, dagState=FAILED
2016-05-18 00:57:43,260 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533062967_0001_1_post
2016-05-18 00:57:43,260 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,260 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:43,260 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:43,260 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8] every/(1/1) ...platform/0_VALUES_REPLACE, with id=dag_1463533062967_0001_1
2016-05-18 00:57:43,260 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8] every/(1/1) ...platform/0_VALUES_REPLACE, with id=dag_1463533062967_0001_1
2016-05-18 00:57:43,268 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...platform/0_VALUES_REPLACE, applicationId=application_1463533062967_0001
2016-05-18 00:57:43,270 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:43,270 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:43,270 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:43,270 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:43,270 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:43,270 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:43,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:43,271 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:43,271 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:43,271 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:43,272 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:43,272 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:43,272 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:43,273 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:43,278 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/2883FCC0A840489EA5A3B72DABE63606/CAE95E8548BA4A559108C9DF92FAF4C8/.tez/application_1463533062967_0001
2016-05-18 00:57:43,279 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:43,279 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...platform/0_VALUES_REPLACE
2016-05-18 00:57:43,279 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:43,279 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_REPLACE/_temporary
2016-05-18 00:57:43,280 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.329
2016-05-18 00:57:43,280 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_VALUES_REPLACE
2016-05-18 00:57:43,305 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:43,305 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:43,336 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 00:57:43,343 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 00:57:43,344 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:43,344 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:43,344 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:43,367 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:43,375 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_SWAP
2016-05-18 00:57:43,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:43,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:43,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_SWAP"]
2016-05-18 00:57:43,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:43,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:43,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:43,423 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...blyplatform/0_VALUES_SWAP
2016-05-18 00:57:43,424 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:43,435 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:43,439 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:43,440 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:43,440 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:43,448 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001 doesn't exist and is created
2016-05-18 00:57:43,458 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:43,458 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:43,462 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:43,462 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:43,462 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001_wd
2016-05-18 00:57:43,472 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533063440_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:43,473 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:43,473 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:43,474 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:43,474 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:43,474 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:43,474 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:43,475 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:43,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:43,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:43,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:43,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:43,476 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:43,477 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:43,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:43,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:43,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:43,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:43,477 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:43,477 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:43,478 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:43,479 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:43,479 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:43,479 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001_wd/localmode-log-dir
2016-05-18 00:57:43,480 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001_wd/localmode-log-dir/history.txt.appattempt_1463533063440_0001_000000, maxErrors=10
2016-05-18 00:57:43,480 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:43,480 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533063440_0001_000000, appSubmitTime=1463533063472, launchTime=1463533063472
2016-05-18 00:57:43,482 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:43,486 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:55265
2016-05-18 00:57:43,487 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533063440
2016-05-18 00:57:43,488 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533063440_0001_000000, startTime=1463533063488
2016-05-18 00:57:43,488 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:43,562 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:43,562 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:43,563 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...blyplatform/0_VALUES_SWAP, applicationId=application_1463533063440_0001, dagName=[E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3] every/(1/1) ...blyplatform/0_VALUES_SWAP
2016-05-18 00:57:43,578 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3] every/(1/1) ...blyplatform/0_VALUES_SWAP
2016-05-18 00:57:43,578 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:43,578 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533063440_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001_wd/localmode-log-dir/dag_1463533063440_0001_1.dot
2016-05-18 00:57:43,579 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001_wd/localmode-log-dir/dag_1463533063440_0001_1-tez-dag.pb.txt
2016-05-18 00:57:43,585 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533063440_0001_1
2016-05-18 00:57:43,585 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,585 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3] every/(1/1) ...blyplatform/0_VALUES_SWAP
2016-05-18 00:57:43 Running Dag: dag_1463533063440_0001_1
2016-05-18 00:57:43,593 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533063440_0001_1, submitTime=1463533063578
2016-05-18 00:57:43,594 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001/tez-conf.pb" } size: 36913 timestamp: 1463533063000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:43,594 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,594 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:43,595 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001/tez-conf.pb" } size: 36913 timestamp: 1463533063000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:43,595 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,595 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:43,595 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:43,596 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533063440_0001_1, initTime=1463533063596
2016-05-18 00:57:43,596 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533063440_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:43,596 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:43,596 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:DAG_STARTED]: dagID=dag_1463533063440_0001_1, startTime=1463533063596
2016-05-18 00:57:43,596 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533063440_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:43,596 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F3F41E0C1957413098CA665BDBB7933C : {63B2050089FF4407BF5B91C043C91994={InputName=63B2050089FF4407BF5B91C043C91994}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:43,596 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 63B2050089FF4407BF5B91C043C91994, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:43,596 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,596 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,596 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,597 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,597 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:43,598 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,599 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 63B2050089FF4407BF5B91C043C91994 on vertex vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,599 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...blyplatform/0_VALUES_SWAP, applicationId=application_1463533063440_0001, dagName=[E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3] every/(1/1) ...blyplatform/0_VALUES_SWAP
2016-05-18 00:57:43,601 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 63B2050089FF4407BF5B91C043C91994 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:43,601 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:43,602 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,602 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,602 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533063440_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:43,602 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4], numAdditionalOutputs=1
2016-05-18 00:57:43,602 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=E639827DBBAC48AE862907440FD926A1, vertex=vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:43,603 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:43,603 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:43,603 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:43,604 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:43,604 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 63B2050089FF4407BF5B91C043C91994 on vertex vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,605 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for FA01AA4DD0384A47B220967D1AC2F1A4:E639827DBBAC48AE862907440FD926A1 using oldmapred API
2016-05-18 00:57:43,605 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=E639827DBBAC48AE862907440FD926A1, vertexName=FA01AA4DD0384A47B220967D1AC2F1A4, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:43,606 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:VERTEX_INITIALIZED]: vertexName=FA01AA4DD0384A47B220967D1AC2F1A4, vertexId=vertex_1463533063440_0001_1_01, initRequestedTime=1463533063598, initedTime=1463533063602, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:43,606 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:43,606 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:43,607 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {63B2050089FF4407BF5B91C043C91994=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:43,607 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] parallelism set to 1
2016-05-18 00:57:43,607 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:43,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F3F41E0C1957413098CA665BDBB7933C, vertexId=vertex_1463533063440_0001_1_00, initRequestedTime=1463533063596, initedTime=1463533063608, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:43,608 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] with distanceFromRoot: 0
2016-05-18 00:57:43,608 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:43,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533063440_0001_1_00, startRequestedTime=1463533063598, startedTime=1463533063608
2016-05-18 00:57:43,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533063440_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:43,608 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:43,608 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533063440_0001_1_00 for vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] numStartedSources: 1 numSources: 1
2016-05-18 00:57:43,608 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] with distanceFromRoot: 1
2016-05-18 00:57:43,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533063440_0001_1_01, startRequestedTime=1463533063608, startedTime=1463533063608
2016-05-18 00:57:43,609 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:43,609 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F3F41E0C1957413098CA665BDBB7933C
2016-05-18 00:57:43,610 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,610 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: FA01AA4DD0384A47B220967D1AC2F1A4 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:43,610 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:TASK_STARTED]: vertexName=F3F41E0C1957413098CA665BDBB7933C, taskId=task_1463533063440_0001_1_00_000000, scheduledTime=1463533063610, launchTime=1463533063610
2016-05-18 00:57:43,612 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F3F41E0C1957413098CA665BDBB7933C in vertex: FA01AA4DD0384A47B220967D1AC2F1A4 numBipartiteSourceTasks: 1
2016-05-18 00:57:43,612 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,613 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:43,615 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533063440_0001_00_000001, launchTime=1463533063615
2016-05-18 00:57:43,616 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533063440_0001_00_000001
2016-05-18 00:57:43,616 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533063440_0001_1_00_000000_0] started. Is using containerId: [container_1463533063440_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:43,616 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F3F41E0C1957413098CA665BDBB7933C, taskAttemptId=attempt_1463533063440_0001_1_00_000000_0, startTime=1463533063616, containerId=container_1463533063440_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:43,617 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533063440_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533063440_0001_1_00_000000_0
2016-05-18 00:57:43 Starting to run new task attempt: attempt_1463533063440_0001_1_00_000000_0
2016-05-18 00:57:43,617 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533063440_0001_1_00_000000_0
2016-05-18 00:57:43,617 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,618 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:43,618 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3] every/(1/1) ...blyplatform/0_VALUES_SWAP, VertexName: F3F41E0C1957413098CA665BDBB7933C, VertexParallelism: 1, TaskAttemptID:attempt_1463533063440_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=63B2050089FF4407BF5B91C043C91994, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=FA01AA4DD0384A47B220967D1AC2F1A4, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:43,618 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:43,618 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533063440_0001_1_00_000000_0
2016-05-18 00:57:43,619 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533063440_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:43,619 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533063440_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:43,620 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533063440_0001, with dag id: dag_1463533063440_0001_1
2016-05-18 00:57:43,623 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:43,625 INFO input.MRInput (MRInput.java:initialize(440)) - 63B2050089FF4407BF5B91C043C91994 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:43,626 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 63B2050089FF4407BF5B91C043C91994
2016-05-18 00:57:43,627 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F3F41E0C1957413098CA665BDBB7933C, ordinal: 0
2016-05-18 00:57:43,627 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F3F41E0C1957413098CA665BDBB7933C, mem on start (mb), free: 101, total: 363, max: 672
2016-05-18 00:57:43,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:43,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:43,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:43,627 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:43,627 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[63B2050089FF4407BF5B91C043C91994:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [FA01AA4DD0384A47B220967D1AC2F1A4:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:43,627 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:43,628 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:43,628 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:43,628 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[63B2050089FF4407BF5B91C043C91994:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [FA01AA4DD0384A47B220967D1AC2F1A4:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:43,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:43,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 63B2050089FF4407BF5B91C043C91994 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:43,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:43,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:43,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 63B2050089FF4407BF5B91C043C91994
2016-05-18 00:57:43,628 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:43,629 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533063440_0001_1_00_000000_0
2016-05-18 00:57:43,630 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:43,630 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 63B2050089FF4407BF5B91C043C91994 initialized RecordReader from event
2016-05-18 00:57:43,631 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:43,632 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 63B2050089FF4407BF5B91C043C91994
2016-05-18 00:57:43,632 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: B030CC1A15AF49C783E09A3673372A27
2016-05-18 00:57:43,632 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] B030CC1A15AF49C783E09A3673372A27
2016-05-18 00:57:43,633 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:43,635 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:43,635 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - FA01AA4DD0384A47B220967D1AC2F1A4 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@541bd039, comparator=cascading.tuple.hadoop.util.TupleComparator@6e037952, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:43,635 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for FA01AA4DD0384A47B220967D1AC2F1A4: , UsingHashComparator=false
2016-05-18 00:57:43,658 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:43,658 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:43,658 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for FA01AA4DD0384A47B220967D1AC2F1A4: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:43,658 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - FA01AA4DD0384A47B220967D1AC2F1A4: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:43,659 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:43,659 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F3F41E0C1957413098CA665BDBB7933C, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:43,661 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533063440_0001_1_00_000000_0
2016-05-18 00:57:43,662 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F3F41E0C1957413098CA665BDBB7933C, mem on close (mb), free: 194, total: 363, max: 672
2016-05-18 00:57:43,662 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - FA01AA4DD0384A47B220967D1AC2F1A4: Starting flush of map output
2016-05-18 00:57:43,662 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - FA01AA4DD0384A47B220967D1AC2F1A4: Span0.length = 20, perItem = 6
2016-05-18 00:57:43,662 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - FA01AA4DD0384A47B220967D1AC2F1A4: done sorting span=0, length=20, time=0
2016-05-18 00:57:43,662 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - FA01AA4DD0384A47B220967D1AC2F1A4: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:43,663 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - FA01AA4DD0384A47B220967D1AC2F1A4: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001_wd/localmode-local-dir/output/attempt_1463533063440_0001_1_00_000000_0_10201_0/file.out
2016-05-18 00:57:43,666 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:43 Completed running task attempt: attempt_1463533063440_0001_1_00_000000_0
2016-05-18 00:57:43,667 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533063440_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:43,667 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533063440_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:43,667 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533063440_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533063659, Process_Duration=2, Process_End_Time=1463533063661, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:43,668 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533063440_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:43,668 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:43,668 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:43,668 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,668 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533063440_0001_00_000001
2016-05-18 00:57:43,668 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F3F41E0C1957413098CA665BDBB7933C, taskAttemptId=attempt_1463533063440_0001_1_00_000000_0, creationTime=1463533063612, allocationTime=1463533063614, startTime=1463533063616, finishTime=1463533063668, timeTaken=52, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:43,669 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533063440_0001_1_00_000000_0
2016-05-18 00:57:43,669 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:TASK_FINISHED]: vertexName=F3F41E0C1957413098CA665BDBB7933C, taskId=task_1463533063440_0001_1_00_000000, startTime=1463533063616, finishTime=1463533063669, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463533063440_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533063659, Process_Duration=2, Process_End_Time=1463533063661, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:43,670 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:43,670 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C]
2016-05-18 00:57:43,671 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:VERTEX_FINISHED]: vertexName=F3F41E0C1957413098CA665BDBB7933C, vertexId=vertex_1463533063440_0001_1_00, initRequestedTime=1463533063596, initedTime=1463533063608, startRequestedTime=1463533063598, startedTime=1463533063608, finishTime=1463533063670, timeTaken=62, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533063659, Process_Duration=2, Process_End_Time=1463533063661, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533063616, firstTasksToStart=[ task_1463533063440_0001_1_00_000000 ], lastTaskFinishTime=1463533063668, lastTasksToFinish=[ task_1463533063440_0001_1_00_000000 ], minTaskDuration=52, maxTaskDuration=52, avgTaskDuration=52.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533063440_0001_1_00_000000 ], longestDurationTasks=[ task_1463533063440_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:43,671 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:43,671 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533063440_0001_00_000001, stoppedTime=1463533063671, exitStatus=0
2016-05-18 00:57:43,671 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] attempt: attempt_1463533063440_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:43,671 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533063440_0001_1_00 [F3F41E0C1957413098CA665BDBB7933C] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:43,671 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: FA01AA4DD0384A47B220967D1AC2F1A4
2016-05-18 00:57:43,671 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F3F41E0C1957413098CA665BDBB7933C, destinationVertexName=FA01AA4DD0384A47B220967D1AC2F1A4. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:43,672 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,671 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:43,672 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533063440_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:43,672 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:TASK_STARTED]: vertexName=FA01AA4DD0384A47B220967D1AC2F1A4, taskId=task_1463533063440_0001_1_01_000000, scheduledTime=1463533063672, launchTime=1463533063672
2016-05-18 00:57:43,672 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,674 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533063440_0001_00_000002, launchTime=1463533063674
2016-05-18 00:57:43,675 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533063440_0001_00_000002
2016-05-18 00:57:43,675 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533063440_0001_1_01_000000_0] started. Is using containerId: [container_1463533063440_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:43,675 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=FA01AA4DD0384A47B220967D1AC2F1A4, taskAttemptId=attempt_1463533063440_0001_1_01_000000_0, startTime=1463533063675, containerId=container_1463533063440_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:43,676 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533063440_0001_00_000002: 1 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533063440_0001_1_01_000000_0
2016-05-18 00:57:43 Starting to run new task attempt: attempt_1463533063440_0001_1_01_000000_0
2016-05-18 00:57:43,676 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533063440_0001_1_01_000000_0
2016-05-18 00:57:43,676 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,676 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:43,676 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3] every/(1/1) ...blyplatform/0_VALUES_SWAP, VertexName: FA01AA4DD0384A47B220967D1AC2F1A4, VertexParallelism: 1, TaskAttemptID:attempt_1463533063440_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F3F41E0C1957413098CA665BDBB7933C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E639827DBBAC48AE862907440FD926A1, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:43,676 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:43,677 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533063440_0001_1_01_000000_0
2016-05-18 00:57:43,679 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:43,680 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:43,680 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533063440_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,680 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533063440_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:43,681 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533063440_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:43,681 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533063440_0001_00_000001, Exiting
2016-05-18 00:57:43,681 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533063440_0001_00_000001
2016-05-18 00:57:43,681 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533063440_0001_00_000001
2016-05-18 00:57:43,681 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533063440_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:43,682 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533063440_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:43,682 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533063440_0001_00_000001 completed successfully
2016-05-18 00:57:43,684 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: FA01AA4DD0384A47B220967D1AC2F1A4, ordinal: 1
2016-05-18 00:57:43,684 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FA01AA4DD0384A47B220967D1AC2F1A4, mem on start (mb), free: 189, total: 363, max: 672
2016-05-18 00:57:43,684 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:43,684 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:43,684 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:43,688 INFO output.MROutput (MROutput.java:initialize(421)) - E639827DBBAC48AE862907440FD926A1: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:43,688 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:43,688 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[E639827DBBAC48AE862907440FD926A1:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [F3F41E0C1957413098CA665BDBB7933C:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:43,688 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:43,688 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:43,688 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:43,688 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[E639827DBBAC48AE862907440FD926A1:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [F3F41E0C1957413098CA665BDBB7933C:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:43,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:43,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F3F41E0C1957413098CA665BDBB7933C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:43,689 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F3F41E0C1957413098CA665BDBB7933C: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:43,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:43,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:43,689 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F3F41E0C1957413098CA665BDBB7933C: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:43,690 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F3F41E0C1957413098CA665BDBB7933C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:43,693 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F3F41E0C1957413098CA665BDBB7933C
2016-05-18 00:57:43,693 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:43,694 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533063440_0001_1_01_000000_0
2016-05-18 00:57:43,694 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F3F41E0C1957413098CA665BDBB7933C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:43,695 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533063440_0001_1_00_000000_0_10201} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533063695, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:43,696 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F3F41E0C1957413098CA665BDBB7933C
2016-05-18 00:57:43,696 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:43,696 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:43,697 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: B030CC1A15AF49C783E09A3673372A27
2016-05-18 00:57:43,697 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_SWAP"], id: E639827DBBAC48AE862907440FD926A1
2016-05-18 00:57:43,697 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_SWAP"]
2016-05-18 00:57:43,697 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F3F41E0C1957413098CA665BDBB7933C, wasInterrupted=false
2016-05-18 00:57:43,697 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] B030CC1A15AF49C783E09A3673372A27, for 1 inputs
2016-05-18 00:57:43,698 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:43,699 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:43,699 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:43,699 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001_wd/localmode-local-dir/output/attempt_1463533063440_0001_1_00_000000_0_10201/file.out, len=181, isLocal=true
2016-05-18 00:57:43,700 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:43,700 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:43,700 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:43,700 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:43,700 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F3F41E0C1957413098CA665BDBB7933C
2016-05-18 00:57:43,700 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: FA01AA4DD0384A47B220967D1AC2F1A4, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:43,700 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F3F41E0C1957413098CA665BDBB7933C: Shuffle Runner thread complete
2016-05-18 00:57:43,701 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:43,701 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F3F41E0C1957413098CA665BDBB7933C: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:43,703 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:43,703 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:43,704 INFO output.MROutput (MROutput.java:close(526)) - E639827DBBAC48AE862907440FD926A1 closed
2016-05-18 00:57:43,704 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533063440_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:43,704 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533063440_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:43,705 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533063440_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533063700, Process_Duration=4, Process_End_Time=1463533063704, Read_Duration=1, Tuples_Read=1]]
2016-05-18 00:57:43,705 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:43,705 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: FA01AA4DD0384A47B220967D1AC2F1A4, mem on close (mb), free: 185, total: 363, max: 672
2016-05-18 00:57:43,705 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=FA01AA4DD0384A47B220967D1AC2F1A4, index=1, interruptedStatus=false
2016-05-18 00:57:43,705 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F3F41E0C1957413098CA665BDBB7933C
2016-05-18 00:57:43,705 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F3F41E0C1957413098CA665BDBB7933C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:43,706 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=FA01AA4DD0384A47B220967D1AC2F1A4, sourceVertex=F3F41E0C1957413098CA665BDBB7933C, interruptedStatus=true
2016-05-18 00:57:43,706 INFO output.MROutput (MROutput.java:close(526)) - E639827DBBAC48AE862907440FD926A1 closed
2016-05-18 00:57:43,706 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=FA01AA4DD0384A47B220967D1AC2F1A4, sourceVertex=E639827DBBAC48AE862907440FD926A1, interruptedStatus=true
2016-05-18 00:57:43,706 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533063440_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:43,706 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:43,706 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533063440_0001_00_000002
2016-05-18 00:57:43,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=FA01AA4DD0384A47B220967D1AC2F1A4, taskAttemptId=attempt_1463533063440_0001_1_01_000000_0, creationTime=1463533063672, allocationTime=1463533063673, startTime=1463533063675, finishTime=1463533063706, timeTaken=31, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533063440_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533063700, Process_Duration=4, Process_End_Time=1463533063704, Read_Duration=1, Tuples_Read=1
2016-05-18 00:57:43,707 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533063440_0001_00_000002
2016-05-18 00:57:43,707 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533063440_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:43,707 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:TASK_FINISHED]: vertexName=FA01AA4DD0384A47B220967D1AC2F1A4, taskId=task_1463533063440_0001_1_01_000000, startTime=1463533063675, finishTime=1463533063707, timeTaken=32, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533063440_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:43,707 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] because task failed: task_1463533063440_0001_1_01_000000
2016-05-18 00:57:43,707 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:43,707 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:43,707 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:43,708 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,708 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=E639827DBBAC48AE862907440FD926A1, vertexId=vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,709 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533063440_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:43,709 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533063440_0001_00_000002 completed but with errors
2016-05-18 00:57:43,709 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:VERTEX_FINISHED]: vertexName=FA01AA4DD0384A47B220967D1AC2F1A4, vertexId=vertex_1463533063440_0001_1_01, initRequestedTime=1463533063598, initedTime=1463533063602, startRequestedTime=1463533063608, startedTime=1463533063608, finishTime=1463533063709, timeTaken=101, status=FAILED, diagnostics=Task failed, taskId=task_1463533063440_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533063440_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533063675, firstTasksToStart=[ task_1463533063440_0001_1_01_000000 ], lastTaskFinishTime=1463533063706, lastTasksToFinish=[ task_1463533063440_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:43,709 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:43,710 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533063440_0001_00_000002, stoppedTime=1463533063710, exitStatus=-1000
2016-05-18 00:57:43,710 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533063440_0001_1_01_000000_0 failed: true container: container_1463533063440_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:43,711 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:43,711 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:43,711 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:43,712 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533063440_0001_1, queueSize=0
2016-05-18 00:57:43,713 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063440_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533063440_0001_1, startTime=1463533063596, finishTime=1463533063712, timeTaken=116, status=FAILED, diagnostics=Vertex failed, vertexName=FA01AA4DD0384A47B220967D1AC2F1A4, vertexId=vertex_1463533063440_0001_1_01, diagnostics=[Task failed, taskId=task_1463533063440_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533063440_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=210, AM_GC_TIME_MILLIS=10, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533063659, Process_Duration=2, Process_End_Time=1463533063661, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:43,713 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533063440_0001_1_01 [FA01AA4DD0384A47B220967D1AC2F1A4]
2016-05-18 00:57:43,713 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533063440_0001_1 finished with state: FAILED
2016-05-18 00:57:43,713 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533063440_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:43,713 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533063440_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:43,714 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533063440_0001_00_000002
2016-05-18 00:57:43 Completed Dag: dag_1463533063440_0001_1
2016-05-18 00:57:43,714 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533063440_0001_1, dagState=FAILED
2016-05-18 00:57:43,714 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533063440_0001_1_post
2016-05-18 00:57:43,714 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:43,714 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 2
2016-05-18 00:57:43,714 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:43,714 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3] every/(1/1) ...blyplatform/0_VALUES_SWAP, with id=dag_1463533063440_0001_1
2016-05-18 00:57:43,714 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3] every/(1/1) ...blyplatform/0_VALUES_SWAP, with id=dag_1463533063440_0001_1
2016-05-18 00:57:43,725 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...blyplatform/0_VALUES_SWAP, applicationId=application_1463533063440_0001
2016-05-18 00:57:43,727 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:43,727 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:43,727 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:43,727 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:43,727 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:43,727 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:43,727 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:43,728 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:43,728 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:43,728 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:43,728 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:43,729 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:43,729 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:43,730 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:43,733 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3AF75EFACDB4A869E41B7DA655966B8/B2C2E54817DD4355AFD993B54A36ECD3/.tez/application_1463533063440_0001
2016-05-18 00:57:43,733 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:43,734 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...blyplatform/0_VALUES_SWAP
2016-05-18 00:57:43,734 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:43,734 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_VALUES_SWAP/_temporary
2016-05-18 00:57:43,735 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.311
2016-05-18 00:57:43,735 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_VALUES_SWAP
2016-05-18 00:57:43,762 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:43,762 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:43,797 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.009
2016-05-18 00:57:43,801 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:43,801 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:43,801 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:43,801 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:43,816 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:43,821 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_-1
2016-05-18 00:57:43,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:43,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:43,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_-1"]
2016-05-18 00:57:43,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:43,865 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:43,866 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:43,866 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...ssemblyplatform/0_ARGS_-1
2016-05-18 00:57:43,866 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:43,878 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:43,883 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:43,883 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:43,883 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:43,892 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001 doesn't exist and is created
2016-05-18 00:57:43,906 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:43,906 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:43,907 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001_wd
2016-05-18 00:57:43,916 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533063883_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:43,916 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:43,916 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:43,917 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:43,917 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:43,917 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:43,917 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:43,918 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:43,919 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:43,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:43,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:43,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:43,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:43,920 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:43,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:43,920 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:43,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:43,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:43,921 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:43,921 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:43,922 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:43,923 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:43,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:43,923 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001_wd/localmode-log-dir
2016-05-18 00:57:43,924 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001_wd/localmode-log-dir/history.txt.appattempt_1463533063883_0001_000000, maxErrors=10
2016-05-18 00:57:43,924 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:43,924 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533063883_0001_000000, appSubmitTime=1463533063915, launchTime=1463533063915
2016-05-18 00:57:43,926 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:43,930 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:47226
2016-05-18 00:57:43,930 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533063883
2016-05-18 00:57:43,932 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533063883_0001_000000, startTime=1463533063932
2016-05-18 00:57:43,932 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:44,006 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:44,006 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:44,006 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...ssemblyplatform/0_ARGS_-1, applicationId=application_1463533063883_0001, dagName=[B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61] every/(1/1) ...ssemblyplatform/0_ARGS_-1
2016-05-18 00:57:44,021 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61] every/(1/1) ...ssemblyplatform/0_ARGS_-1
2016-05-18 00:57:44,022 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:44,022 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533063883_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001_wd/localmode-log-dir/dag_1463533063883_0001_1.dot
2016-05-18 00:57:44,023 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001_wd/localmode-log-dir/dag_1463533063883_0001_1-tez-dag.pb.txt
2016-05-18 00:57:44,026 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:44,026 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:44,032 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533063883_0001_1
2016-05-18 00:57:44,032 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,032 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61] every/(1/1) ...ssemblyplatform/0_ARGS_-1
2016-05-18 00:57:44 Running Dag: dag_1463533063883_0001_1
2016-05-18 00:57:44,042 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533063883_0001_1, submitTime=1463533064021
2016-05-18 00:57:44,043 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001/tez-conf.pb" } size: 36913 timestamp: 1463533063000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:44,043 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,044 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:44,045 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001/tez-conf.pb" } size: 36913 timestamp: 1463533063000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:44,046 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]
2016-05-18 00:57:44,046 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:44,047 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:44,048 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533063883_0001_1, initTime=1463533064048
2016-05-18 00:57:44,048 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533063883_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:44,048 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:44,049 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:DAG_STARTED]: dagID=dag_1463533063883_0001_1, startTime=1463533064049
2016-05-18 00:57:44,049 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533063883_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:44,049 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B5154B0110D04360B385BAE2DFEFA696 : {988281FAD515427AB87D5D94DDDFE1AC={InputName=988281FAD515427AB87D5D94DDDFE1AC}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:44,049 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 988281FAD515427AB87D5D94DDDFE1AC, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:44,049 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,049 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,049 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,049 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,050 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:44,050 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 988281FAD515427AB87D5D94DDDFE1AC on vertex vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,050 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]
2016-05-18 00:57:44,050 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...ssemblyplatform/0_ARGS_-1, applicationId=application_1463533063883_0001, dagName=[B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61] every/(1/1) ...ssemblyplatform/0_ARGS_-1
2016-05-18 00:57:44,053 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:44,053 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]
2016-05-18 00:57:44,053 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]
2016-05-18 00:57:44,054 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 988281FAD515427AB87D5D94DDDFE1AC asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:44,054 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533063883_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:44,054 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2], numAdditionalOutputs=1
2016-05-18 00:57:44,054 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=EAA3670C620240948845E84FDC21C6CE, vertex=vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:44,056 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:44,056 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:44,057 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for C7190809AA9D4B0D87CBF962EED948D2:EAA3670C620240948845E84FDC21C6CE using oldmapred API
2016-05-18 00:57:44,057 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=EAA3670C620240948845E84FDC21C6CE, vertexName=C7190809AA9D4B0D87CBF962EED948D2, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:44,057 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:44,057 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:44,058 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 988281FAD515427AB87D5D94DDDFE1AC on vertex vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,058 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C7190809AA9D4B0D87CBF962EED948D2, vertexId=vertex_1463533063883_0001_1_01, initRequestedTime=1463533064050, initedTime=1463533064054, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:44,058 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:44,059 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:44,059 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {988281FAD515427AB87D5D94DDDFE1AC=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:44,059 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] parallelism set to 1
2016-05-18 00:57:44,059 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:44,060 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B5154B0110D04360B385BAE2DFEFA696, vertexId=vertex_1463533063883_0001_1_00, initRequestedTime=1463533064049, initedTime=1463533064059, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:44,060 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] with distanceFromRoot: 0
2016-05-18 00:57:44,060 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:44,060 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533063883_0001_1_00, startRequestedTime=1463533064050, startedTime=1463533064060
2016-05-18 00:57:44,060 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533063883_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:44,060 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:44,060 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533063883_0001_1_00 for vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2] numStartedSources: 1 numSources: 1
2016-05-18 00:57:44,060 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2] with distanceFromRoot: 1
2016-05-18 00:57:44,061 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533063883_0001_1_01, startRequestedTime=1463533064060, startedTime=1463533064060
2016-05-18 00:57:44,061 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:44,062 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: C7190809AA9D4B0D87CBF962EED948D2 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:44,062 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B5154B0110D04360B385BAE2DFEFA696
2016-05-18 00:57:44,063 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B5154B0110D04360B385BAE2DFEFA696 in vertex: C7190809AA9D4B0D87CBF962EED948D2 numBipartiteSourceTasks: 1
2016-05-18 00:57:44,063 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,064 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:TASK_STARTED]: vertexName=B5154B0110D04360B385BAE2DFEFA696, taskId=task_1463533063883_0001_1_00_000000, scheduledTime=1463533064064, launchTime=1463533064064
2016-05-18 00:57:44,064 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,066 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:44,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533063883_0001_00_000001, launchTime=1463533064067
2016-05-18 00:57:44,068 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533063883_0001_00_000001
2016-05-18 00:57:44,069 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533063883_0001_1_00_000000_0] started. Is using containerId: [container_1463533063883_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:44,069 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B5154B0110D04360B385BAE2DFEFA696, taskAttemptId=attempt_1463533063883_0001_1_00_000000_0, startTime=1463533064069, containerId=container_1463533063883_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:44,069 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533063883_0001, with dag id: dag_1463533063883_0001_1
2016-05-18 00:57:44,069 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533063883_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533063883_0001_1_00_000000_0
2016-05-18 00:57:44 Starting to run new task attempt: attempt_1463533063883_0001_1_00_000000_0
2016-05-18 00:57:44,070 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533063883_0001_1_00_000000_0
2016-05-18 00:57:44,070 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,070 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:44,071 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61] every/(1/1) ...ssemblyplatform/0_ARGS_-1, VertexName: B5154B0110D04360B385BAE2DFEFA696, VertexParallelism: 1, TaskAttemptID:attempt_1463533063883_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=988281FAD515427AB87D5D94DDDFE1AC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C7190809AA9D4B0D87CBF962EED948D2, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:44,071 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:44,071 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533063883_0001_1_00_000000_0
2016-05-18 00:57:44,073 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533063883_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:44,073 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533063883_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:44,076 INFO input.MRInput (MRInput.java:initialize(440)) - 988281FAD515427AB87D5D94DDDFE1AC using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:44,076 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 988281FAD515427AB87D5D94DDDFE1AC
2016-05-18 00:57:44,076 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:44,079 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B5154B0110D04360B385BAE2DFEFA696, ordinal: 0
2016-05-18 00:57:44,080 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B5154B0110D04360B385BAE2DFEFA696, mem on start (mb), free: 101, total: 363, max: 672
2016-05-18 00:57:44,080 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:44,080 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:44,080 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:44,080 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:44,080 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[988281FAD515427AB87D5D94DDDFE1AC:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [C7190809AA9D4B0D87CBF962EED948D2:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:44,080 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:44,080 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:44,081 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:44,081 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[988281FAD515427AB87D5D94DDDFE1AC:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [C7190809AA9D4B0D87CBF962EED948D2:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:44,081 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:44,081 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 988281FAD515427AB87D5D94DDDFE1AC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:44,081 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:44,081 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 988281FAD515427AB87D5D94DDDFE1AC
2016-05-18 00:57:44,081 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:44,082 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:44,082 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533063883_0001_1_00_000000_0
2016-05-18 00:57:44,083 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:44,083 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 988281FAD515427AB87D5D94DDDFE1AC initialized RecordReader from event
2016-05-18 00:57:44,086 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:44,086 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 988281FAD515427AB87D5D94DDDFE1AC
2016-05-18 00:57:44,086 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: D806DB63C5744BF1A4E49B18041E6D55
2016-05-18 00:57:44,086 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] D806DB63C5744BF1A4E49B18041E6D55
2016-05-18 00:57:44,088 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:44,090 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:44,090 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - C7190809AA9D4B0D87CBF962EED948D2 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@64ae9884, comparator=cascading.tuple.hadoop.util.TupleComparator@29f35c0, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:44,090 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for C7190809AA9D4B0D87CBF962EED948D2: , UsingHashComparator=false
2016-05-18 00:57:44,112 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:44,112 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:44,113 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for C7190809AA9D4B0D87CBF962EED948D2: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:44,113 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - C7190809AA9D4B0D87CBF962EED948D2: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:44,113 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:44,113 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B5154B0110D04360B385BAE2DFEFA696, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:44,118 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533063883_0001_1_00_000000_0
2016-05-18 00:57:44,118 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B5154B0110D04360B385BAE2DFEFA696, mem on close (mb), free: 194, total: 365, max: 672
2016-05-18 00:57:44,118 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - C7190809AA9D4B0D87CBF962EED948D2: Starting flush of map output
2016-05-18 00:57:44,118 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - C7190809AA9D4B0D87CBF962EED948D2: Span0.length = 20, perItem = 6
2016-05-18 00:57:44,118 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - C7190809AA9D4B0D87CBF962EED948D2: done sorting span=0, length=20, time=0
2016-05-18 00:57:44,118 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - C7190809AA9D4B0D87CBF962EED948D2: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:44,119 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - C7190809AA9D4B0D87CBF962EED948D2: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001_wd/localmode-local-dir/output/attempt_1463533063883_0001_1_00_000000_0_10207_0/file.out
2016-05-18 00:57:44,123 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:44 Completed running task attempt: attempt_1463533063883_0001_1_00_000000_0
2016-05-18 00:57:44,123 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533063883_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:44,123 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533063883_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:44,123 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533063883_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533064114, Process_Duration=4, Process_End_Time=1463533064118, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:44,124 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533063883_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:44,124 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:44,124 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:44,124 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,124 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533063883_0001_00_000001
2016-05-18 00:57:44,124 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B5154B0110D04360B385BAE2DFEFA696, taskAttemptId=attempt_1463533063883_0001_1_00_000000_0, creationTime=1463533064064, allocationTime=1463533064067, startTime=1463533064069, finishTime=1463533064124, timeTaken=55, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:44,125 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533063883_0001_1_00_000000_0
2016-05-18 00:57:44,125 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:TASK_FINISHED]: vertexName=B5154B0110D04360B385BAE2DFEFA696, taskId=task_1463533063883_0001_1_00_000000, startTime=1463533064069, finishTime=1463533064125, timeTaken=56, status=SUCCEEDED, successfulAttemptID=attempt_1463533063883_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533064114, Process_Duration=4, Process_End_Time=1463533064118, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:44,126 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:44,126 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,126 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:VERTEX_FINISHED]: vertexName=B5154B0110D04360B385BAE2DFEFA696, vertexId=vertex_1463533063883_0001_1_00, initRequestedTime=1463533064049, initedTime=1463533064059, startRequestedTime=1463533064050, startedTime=1463533064060, finishTime=1463533064126, timeTaken=66, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533064114, Process_Duration=4, Process_End_Time=1463533064118, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533064069, firstTasksToStart=[ task_1463533063883_0001_1_00_000000 ], lastTaskFinishTime=1463533064124, lastTasksToFinish=[ task_1463533063883_0001_1_00_000000 ], minTaskDuration=55, maxTaskDuration=55, avgTaskDuration=55.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533063883_0001_1_00_000000 ], longestDurationTasks=[ task_1463533063883_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:44,126 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:44,127 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533063883_0001_00_000001, stoppedTime=1463533064127, exitStatus=0
2016-05-18 00:57:44,127 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2] attempt: attempt_1463533063883_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:44,127 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:44,127 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:44,128 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C7190809AA9D4B0D87CBF962EED948D2
2016-05-18 00:57:44,128 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B5154B0110D04360B385BAE2DFEFA696, destinationVertexName=C7190809AA9D4B0D87CBF962EED948D2. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:44,128 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]
2016-05-18 00:57:44,129 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:TASK_STARTED]: vertexName=C7190809AA9D4B0D87CBF962EED948D2, taskId=task_1463533063883_0001_1_01_000000, scheduledTime=1463533064129, launchTime=1463533064129
2016-05-18 00:57:44,129 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]
2016-05-18 00:57:44,130 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533063883_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:44,132 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533063883_0001_00_000002, launchTime=1463533064132
2016-05-18 00:57:44,133 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533063883_0001_00_000002
2016-05-18 00:57:44,133 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533063883_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533063883_0001_1_01_000000_0
2016-05-18 00:57:44,133 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533063883_0001_1_01_000000_0] started. Is using containerId: [container_1463533063883_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:44,133 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C7190809AA9D4B0D87CBF962EED948D2, taskAttemptId=attempt_1463533063883_0001_1_01_000000_0, startTime=1463533064133, containerId=container_1463533063883_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:44 Starting to run new task attempt: attempt_1463533063883_0001_1_01_000000_0
2016-05-18 00:57:44,134 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533063883_0001_1_01_000000_0
2016-05-18 00:57:44,134 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,134 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:44,134 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61] every/(1/1) ...ssemblyplatform/0_ARGS_-1, VertexName: C7190809AA9D4B0D87CBF962EED948D2, VertexParallelism: 1, TaskAttemptID:attempt_1463533063883_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B5154B0110D04360B385BAE2DFEFA696, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EAA3670C620240948845E84FDC21C6CE, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:44,135 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533063883_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:44,135 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533063883_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:44,135 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533063883_0001_00_000001, Exiting
2016-05-18 00:57:44,135 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533063883_0001_00_000001
2016-05-18 00:57:44,135 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533063883_0001_00_000001
2016-05-18 00:57:44,135 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:44,135 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533063883_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:44,136 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533063883_0001_1_01_000000_0
2016-05-18 00:57:44,136 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533063883_0001_00_000001 completed successfully
2016-05-18 00:57:44,138 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:44,139 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533063883_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]
2016-05-18 00:57:44,139 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533063883_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:44,139 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:44,143 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C7190809AA9D4B0D87CBF962EED948D2, ordinal: 1
2016-05-18 00:57:44,143 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C7190809AA9D4B0D87CBF962EED948D2, mem on start (mb), free: 188, total: 365, max: 672
2016-05-18 00:57:44,143 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:44,143 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:44,143 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:44,145 INFO output.MROutput (MROutput.java:initialize(421)) - EAA3670C620240948845E84FDC21C6CE: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:44,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:44,145 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[EAA3670C620240948845E84FDC21C6CE:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [B5154B0110D04360B385BAE2DFEFA696:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:44,145 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:44,145 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:44,145 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:44,145 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[EAA3670C620240948845E84FDC21C6CE:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [B5154B0110D04360B385BAE2DFEFA696:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:44,146 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:44,146 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B5154B0110D04360B385BAE2DFEFA696 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:44,146 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:44,147 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B5154B0110D04360B385BAE2DFEFA696: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:44,148 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B5154B0110D04360B385BAE2DFEFA696: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:44,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:44,149 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B5154B0110D04360B385BAE2DFEFA696 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:44,150 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B5154B0110D04360B385BAE2DFEFA696
2016-05-18 00:57:44,150 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:44,150 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533063883_0001_1_01_000000_0
2016-05-18 00:57:44,152 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B5154B0110D04360B385BAE2DFEFA696: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:44,152 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:44,153 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533063883_0001_1_00_000000_0_10207} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533064153, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:44,153 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B5154B0110D04360B385BAE2DFEFA696
2016-05-18 00:57:44,153 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: D806DB63C5744BF1A4E49B18041E6D55
2016-05-18 00:57:44,153 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_-1"], id: EAA3670C620240948845E84FDC21C6CE
2016-05-18 00:57:44,153 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:44,154 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_-1"]
2016-05-18 00:57:44,154 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] D806DB63C5744BF1A4E49B18041E6D55, for 1 inputs
2016-05-18 00:57:44,154 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B5154B0110D04360B385BAE2DFEFA696, wasInterrupted=false
2016-05-18 00:57:44,154 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:44,155 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:44,156 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:44,156 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001_wd/localmode-local-dir/output/attempt_1463533063883_0001_1_00_000000_0_10207/file.out, len=181, isLocal=true
2016-05-18 00:57:44,156 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:44,157 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:44,157 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:44,157 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:44,157 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C7190809AA9D4B0D87CBF962EED948D2, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:44,157 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B5154B0110D04360B385BAE2DFEFA696
2016-05-18 00:57:44,158 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:44,158 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B5154B0110D04360B385BAE2DFEFA696: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:44,159 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B5154B0110D04360B385BAE2DFEFA696: Shuffle Runner thread complete
2016-05-18 00:57:44,163 INFO output.MROutput (MROutput.java:close(526)) - EAA3670C620240948845E84FDC21C6CE closed
2016-05-18 00:57:44,163 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533063883_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:44,164 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533063883_0001_1_01_000000_0
2016-05-18 00:57:44,164 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C7190809AA9D4B0D87CBF962EED948D2, mem on close (mb), free: 185, total: 365, max: 672
2016-05-18 00:57:44,164 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B5154B0110D04360B385BAE2DFEFA696
2016-05-18 00:57:44,164 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B5154B0110D04360B385BAE2DFEFA696: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:44,164 INFO output.MROutput (MROutput.java:close(526)) - EAA3670C620240948845E84FDC21C6CE closed
2016-05-18 00:57:44,164 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:44 Completed running task attempt: attempt_1463533063883_0001_1_01_000000_0
2016-05-18 00:57:44,164 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533063883_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:44,165 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533063883_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:44,165 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533063883_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533064157, Process_Duration=7, Process_End_Time=1463533064164, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:44,165 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533063883_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:44,166 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:44,166 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C7190809AA9D4B0D87CBF962EED948D2, taskAttemptId=attempt_1463533063883_0001_1_01_000000_0, creationTime=1463533064129, allocationTime=1463533064130, startTime=1463533064133, finishTime=1463533064166, timeTaken=33, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:44,166 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533063883_0001_1_01_000000_0
2016-05-18 00:57:44,166 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:44,166 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,166 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533063883_0001_00_000002
2016-05-18 00:57:44,166 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:TASK_FINISHED]: vertexName=C7190809AA9D4B0D87CBF962EED948D2, taskId=task_1463533063883_0001_1_01_000000, startTime=1463533064133, finishTime=1463533064166, timeTaken=33, status=SUCCEEDED, successfulAttemptID=attempt_1463533063883_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533064157, Process_Duration=7, Process_End_Time=1463533064164, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:44,167 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:44,167 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2]
2016-05-18 00:57:44,167 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:VERTEX_FINISHED]: vertexName=C7190809AA9D4B0D87CBF962EED948D2, vertexId=vertex_1463533063883_0001_1_01, initRequestedTime=1463533064050, initedTime=1463533064054, startRequestedTime=1463533064060, startedTime=1463533064060, finishTime=1463533064167, timeTaken=107, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533064157, Process_Duration=7, Process_End_Time=1463533064164, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533064133, firstTasksToStart=[ task_1463533063883_0001_1_01_000000 ], lastTaskFinishTime=1463533064166, lastTasksToFinish=[ task_1463533063883_0001_1_01_000000 ], minTaskDuration=33, maxTaskDuration=33, avgTaskDuration=33.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533063883_0001_1_01_000000 ], longestDurationTasks=[ task_1463533063883_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:44,168 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:44,168 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533063883_0001_00_000002, stoppedTime=1463533064168, exitStatus=0
2016-05-18 00:57:44,168 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:44,168 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:44,168 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533063883_0001_1_00 [B5154B0110D04360B385BAE2DFEFA696]
2016-05-18 00:57:44,168 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533063883_0001_1
2016-05-18 00:57:44,168 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533063883_0001_1
2016-05-18 00:57:44,168 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533063883_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:44,168 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: EAA3670C620240948845E84FDC21C6CE for vertex: vertex_1463533063883_0001_1_01 [C7190809AA9D4B0D87CBF962EED948D2], outputName: EAA3670C620240948845E84FDC21C6CE
2016-05-18 00:57:44,169 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533063883_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:44,171 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533063883_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:44,171 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533063883_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:44,171 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533063883_0001_00_000002, Exiting
2016-05-18 00:57:44,171 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533063883_0001_00_000002
2016-05-18 00:57:44,172 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533063883_0001_00_000002
2016-05-18 00:57:44,172 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533063883_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:44,172 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533063883_0001_00_000002 completed successfully
2016-05-18 00:57:44,175 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:EAA3670C620240948845E84FDC21C6CE of vertex/vertexGroup:C7190809AA9D4B0D87CBF962EED948D2 isVertexGroupOutput:false
2016-05-18 00:57:44,175 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:44,176 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533063883_0001_1, queueSize=0
2016-05-18 00:57:44,176 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533063883_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533063883_0001_1, startTime=1463533064048, finishTime=1463533064175, timeTaken=127, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=230, AM_GC_TIME_MILLIS=9, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066128271, Process_Duration=11, Process_End_Time=2927066128282, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:44,177 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533063883_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:44,177 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533063883_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:44 Completed Dag: dag_1463533063883_0001_1
2016-05-18 00:57:44,177 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533063883_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:44,177 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533063883_0001_1_post
2016-05-18 00:57:44,177 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,177 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:44,177 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:44,177 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61] every/(1/1) ...ssemblyplatform/0_ARGS_-1, with id=dag_1463533063883_0001_1
2016-05-18 00:57:44,177 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61] every/(1/1) ...ssemblyplatform/0_ARGS_-1, with id=dag_1463533063883_0001_1
2016-05-18 00:57:44,185 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...ssemblyplatform/0_ARGS_-1, applicationId=application_1463533063883_0001
2016-05-18 00:57:44,187 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:44,187 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:44,187 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:44,187 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:44,187 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:44,187 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:44,187 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:44,188 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:44,188 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:44,188 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:44,188 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:44,188 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:44,189 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:44,189 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:44,192 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/B60440DC2BC645EDA3CE0EA69A11727B/DCB993C5CA824C5CA73510523E342D61/.tez/application_1463533063883_0001
2016-05-18 00:57:44,193 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_-1/_temporary
2016-05-18 00:57:44,193 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.327
2016-05-18 00:57:44,260 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_ARGS_-1
2016-05-18 00:57:44,283 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:44,283 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:44,313 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.005
2016-05-18 00:57:44,313 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.005
2016-05-18 00:57:44,313 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:44,314 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:44,314 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_ARGS_'field'
2016-05-18 00:57:44,337 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:44,338 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:44,368 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:44,379 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 00:57:44,379 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:44,379 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:44,380 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:44,393 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:44,397 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_RESULTS
2016-05-18 00:57:44,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:44,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:44,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_RESULTS"]
2016-05-18 00:57:44,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:44,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:44,442 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:44,442 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/0_ARGS_RESULTS
2016-05-18 00:57:44,443 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:44,459 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:44,463 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:44,464 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:44,464 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:44,472 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001 doesn't exist and is created
2016-05-18 00:57:44,487 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:44,487 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:44,487 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001_wd
2016-05-18 00:57:44,498 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533064464_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:44,499 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:44,499 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:44,500 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:44,500 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:44,501 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:44,501 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:44,502 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:44,503 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:44,503 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:44,503 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:44,504 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:44,504 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:44,504 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:44,505 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:44,505 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:44,505 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:44,505 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:44,506 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:44,506 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:44,508 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:44,508 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:44,508 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:44,509 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001_wd/localmode-log-dir
2016-05-18 00:57:44,509 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001_wd/localmode-log-dir/history.txt.appattempt_1463533064464_0001_000000, maxErrors=10
2016-05-18 00:57:44,509 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:44,509 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533064464_0001_000000, appSubmitTime=1463533064498, launchTime=1463533064498
2016-05-18 00:57:44,512 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:44,514 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:58618
2016-05-18 00:57:44,518 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533064464
2016-05-18 00:57:44,519 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533064464_0001_000000, startTime=1463533064519
2016-05-18 00:57:44,519 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:44,587 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:44,587 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:44,587 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/0_ARGS_RESULTS, applicationId=application_1463533064464_0001, dagName=[E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD] every/(1/1) ...lyplatform/0_ARGS_RESULTS
2016-05-18 00:57:44,610 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD] every/(1/1) ...lyplatform/0_ARGS_RESULTS
2016-05-18 00:57:44,610 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:44,611 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533064464_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001_wd/localmode-log-dir/dag_1463533064464_0001_1.dot
2016-05-18 00:57:44,611 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001_wd/localmode-log-dir/dag_1463533064464_0001_1-tez-dag.pb.txt
2016-05-18 00:57:44,618 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533064464_0001_1
2016-05-18 00:57:44,618 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,618 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD] every/(1/1) ...lyplatform/0_ARGS_RESULTS
2016-05-18 00:57:44 Running Dag: dag_1463533064464_0001_1
2016-05-18 00:57:44,627 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533064464_0001_1, submitTime=1463533064610
2016-05-18 00:57:44,628 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001/tez-conf.pb" } size: 36913 timestamp: 1463533064000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:44,629 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,629 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:44,630 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001/tez-conf.pb" } size: 36913 timestamp: 1463533064000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:44,630 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]
2016-05-18 00:57:44,631 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:44,632 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:44,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533064464_0001_1, initTime=1463533064632
2016-05-18 00:57:44,632 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533064464_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:44,632 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:44,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:DAG_STARTED]: dagID=dag_1463533064464_0001_1, startTime=1463533064632
2016-05-18 00:57:44,632 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533064464_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:44,632 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: BC757255BE9E4BB78D5C53E9FF951358 : {020AE1B5EDCD4A3087DD33919C9159B4={InputName=020AE1B5EDCD4A3087DD33919C9159B4}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:44,633 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 020AE1B5EDCD4A3087DD33919C9159B4, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:44,633 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,633 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,633 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,633 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,633 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:44,633 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]
2016-05-18 00:57:44,636 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 020AE1B5EDCD4A3087DD33919C9159B4 on vertex vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,637 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:44,637 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]
2016-05-18 00:57:44,637 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]
2016-05-18 00:57:44,637 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533064464_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:44,637 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D], numAdditionalOutputs=1
2016-05-18 00:57:44,637 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=222E6B4F247A48B2B17F679616223290, vertex=vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:44,637 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 020AE1B5EDCD4A3087DD33919C9159B4 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:44,639 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:44,639 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:44,639 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for CF72614CA9B24AB6A85EDF59E712BA1D:222E6B4F247A48B2B17F679616223290 using oldmapred API
2016-05-18 00:57:44,639 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=222E6B4F247A48B2B17F679616223290, vertexName=CF72614CA9B24AB6A85EDF59E712BA1D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:44,639 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:44,640 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:44,640 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CF72614CA9B24AB6A85EDF59E712BA1D, vertexId=vertex_1463533064464_0001_1_01, initRequestedTime=1463533064633, initedTime=1463533064637, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:44,640 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:44,640 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/0_ARGS_RESULTS, applicationId=application_1463533064464_0001, dagName=[E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD] every/(1/1) ...lyplatform/0_ARGS_RESULTS
2016-05-18 00:57:44,641 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 020AE1B5EDCD4A3087DD33919C9159B4 on vertex vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,641 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:44,642 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {020AE1B5EDCD4A3087DD33919C9159B4=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:44,642 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] parallelism set to 1
2016-05-18 00:57:44,642 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:44,643 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BC757255BE9E4BB78D5C53E9FF951358, vertexId=vertex_1463533064464_0001_1_00, initRequestedTime=1463533064632, initedTime=1463533064643, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:44,643 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] with distanceFromRoot: 0
2016-05-18 00:57:44,643 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:44,643 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533064464_0001_1_00, startRequestedTime=1463533064633, startedTime=1463533064643
2016-05-18 00:57:44,643 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533064464_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:44,644 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:44,644 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533064464_0001_1_00 for vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D] numStartedSources: 1 numSources: 1
2016-05-18 00:57:44,644 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D] with distanceFromRoot: 1
2016-05-18 00:57:44,644 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in BC757255BE9E4BB78D5C53E9FF951358
2016-05-18 00:57:44,645 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,645 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533064464_0001_1_01, startRequestedTime=1463533064644, startedTime=1463533064644
2016-05-18 00:57:44,645 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:44,646 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:TASK_STARTED]: vertexName=BC757255BE9E4BB78D5C53E9FF951358, taskId=task_1463533064464_0001_1_00_000000, scheduledTime=1463533064645, launchTime=1463533064645
2016-05-18 00:57:44,646 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,647 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:44,649 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533064464_0001_00_000001, launchTime=1463533064649
2016-05-18 00:57:44,650 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533064464_0001_00_000001
2016-05-18 00:57:44,651 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533064464_0001_1_00_000000_0] started. Is using containerId: [container_1463533064464_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:44,651 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BC757255BE9E4BB78D5C53E9FF951358, taskAttemptId=attempt_1463533064464_0001_1_00_000000_0, startTime=1463533064651, containerId=container_1463533064464_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:44,651 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: CF72614CA9B24AB6A85EDF59E712BA1D with 0 source tasks and 1 pending tasks
2016-05-18 00:57:44,653 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BC757255BE9E4BB78D5C53E9FF951358 in vertex: CF72614CA9B24AB6A85EDF59E712BA1D numBipartiteSourceTasks: 1
2016-05-18 00:57:44,653 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533064464_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533064464_0001_1_00_000000_0
2016-05-18 00:57:44,653 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:44,654 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:44 Starting to run new task attempt: attempt_1463533064464_0001_1_00_000000_0
2016-05-18 00:57:44,654 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533064464_0001_1_00_000000_0
2016-05-18 00:57:44,654 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,655 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:44,655 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD] every/(1/1) ...lyplatform/0_ARGS_RESULTS, VertexName: BC757255BE9E4BB78D5C53E9FF951358, VertexParallelism: 1, TaskAttemptID:attempt_1463533064464_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=020AE1B5EDCD4A3087DD33919C9159B4, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CF72614CA9B24AB6A85EDF59E712BA1D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:44,655 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:44,655 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533064464_0001_1_00_000000_0
2016-05-18 00:57:44,656 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533064464_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:44,656 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533064464_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:44,660 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:44,661 INFO input.MRInput (MRInput.java:initialize(440)) - 020AE1B5EDCD4A3087DD33919C9159B4 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:44,661 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 020AE1B5EDCD4A3087DD33919C9159B4
2016-05-18 00:57:44,664 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BC757255BE9E4BB78D5C53E9FF951358, ordinal: 0
2016-05-18 00:57:44,664 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BC757255BE9E4BB78D5C53E9FF951358, mem on start (mb), free: 278, total: 365, max: 672
2016-05-18 00:57:44,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:44,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:44,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:44,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:44,664 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[020AE1B5EDCD4A3087DD33919C9159B4:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [CF72614CA9B24AB6A85EDF59E712BA1D:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:44,664 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:44,664 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:44,665 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:44,665 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[020AE1B5EDCD4A3087DD33919C9159B4:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [CF72614CA9B24AB6A85EDF59E712BA1D:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:44,665 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:44,665 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 020AE1B5EDCD4A3087DD33919C9159B4 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:44,665 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:44,665 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:44,665 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 020AE1B5EDCD4A3087DD33919C9159B4
2016-05-18 00:57:44,666 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:44,666 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533064464_0001_1_00_000000_0
2016-05-18 00:57:44,669 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:44,670 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 020AE1B5EDCD4A3087DD33919C9159B4
2016-05-18 00:57:44,670 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: F6BEDA400CC642ABA666C1060A266CB9
2016-05-18 00:57:44,670 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] F6BEDA400CC642ABA666C1060A266CB9
2016-05-18 00:57:44,672 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:44,673 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:44,673 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 020AE1B5EDCD4A3087DD33919C9159B4 initialized RecordReader from event
2016-05-18 00:57:44,675 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:44,675 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CF72614CA9B24AB6A85EDF59E712BA1D using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@7a6480a7, comparator=cascading.tuple.hadoop.util.TupleComparator@397d4303, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:44,676 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CF72614CA9B24AB6A85EDF59E712BA1D: , UsingHashComparator=false
2016-05-18 00:57:44,677 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533064464_0001, with dag id: dag_1463533064464_0001_1
2016-05-18 00:57:44,689 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:44,689 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:44,690 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CF72614CA9B24AB6A85EDF59E712BA1D: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:44,690 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CF72614CA9B24AB6A85EDF59E712BA1D: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:44,690 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:44,691 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BC757255BE9E4BB78D5C53E9FF951358, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:44,693 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533064464_0001_1_00_000000_0
2016-05-18 00:57:44,693 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BC757255BE9E4BB78D5C53E9FF951358, mem on close (mb), free: 173, total: 365, max: 672
2016-05-18 00:57:44,693 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CF72614CA9B24AB6A85EDF59E712BA1D: Starting flush of map output
2016-05-18 00:57:44,694 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CF72614CA9B24AB6A85EDF59E712BA1D: Span0.length = 20, perItem = 6
2016-05-18 00:57:44,694 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CF72614CA9B24AB6A85EDF59E712BA1D: done sorting span=0, length=20, time=0
2016-05-18 00:57:44,694 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CF72614CA9B24AB6A85EDF59E712BA1D: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:44,695 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CF72614CA9B24AB6A85EDF59E712BA1D: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001_wd/localmode-local-dir/output/attempt_1463533064464_0001_1_00_000000_0_10213_0/file.out
2016-05-18 00:57:44,699 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:44 Completed running task attempt: attempt_1463533064464_0001_1_00_000000_0
2016-05-18 00:57:44,699 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533064464_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:44,699 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533064464_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:44,700 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533064464_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533064691, Process_Duration=2, Process_End_Time=1463533064693, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:44,700 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533064464_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:44,700 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:44,700 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BC757255BE9E4BB78D5C53E9FF951358, taskAttemptId=attempt_1463533064464_0001_1_00_000000_0, creationTime=1463533064646, allocationTime=1463533064648, startTime=1463533064651, finishTime=1463533064700, timeTaken=49, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:44,700 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:44,701 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,701 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533064464_0001_1_00_000000_0
2016-05-18 00:57:44,701 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533064464_0001_00_000001
2016-05-18 00:57:44,702 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:TASK_FINISHED]: vertexName=BC757255BE9E4BB78D5C53E9FF951358, taskId=task_1463533064464_0001_1_00_000000, startTime=1463533064651, finishTime=1463533064701, timeTaken=50, status=SUCCEEDED, successfulAttemptID=attempt_1463533064464_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533064691, Process_Duration=2, Process_End_Time=1463533064693, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:44,702 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:44,702 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,703 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:VERTEX_FINISHED]: vertexName=BC757255BE9E4BB78D5C53E9FF951358, vertexId=vertex_1463533064464_0001_1_00, initRequestedTime=1463533064632, initedTime=1463533064643, startRequestedTime=1463533064633, startedTime=1463533064643, finishTime=1463533064702, timeTaken=59, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533064691, Process_Duration=2, Process_End_Time=1463533064693, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533064651, firstTasksToStart=[ task_1463533064464_0001_1_00_000000 ], lastTaskFinishTime=1463533064700, lastTasksToFinish=[ task_1463533064464_0001_1_00_000000 ], minTaskDuration=49, maxTaskDuration=49, avgTaskDuration=49.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533064464_0001_1_00_000000 ], longestDurationTasks=[ task_1463533064464_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:44,703 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:44,703 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533064464_0001_00_000001, stoppedTime=1463533064703, exitStatus=0
2016-05-18 00:57:44,703 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D] attempt: attempt_1463533064464_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:44,703 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:44,703 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CF72614CA9B24AB6A85EDF59E712BA1D
2016-05-18 00:57:44,703 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:44,703 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BC757255BE9E4BB78D5C53E9FF951358, destinationVertexName=CF72614CA9B24AB6A85EDF59E712BA1D. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:44,704 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533064464_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:44,704 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]
2016-05-18 00:57:44,704 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:TASK_STARTED]: vertexName=CF72614CA9B24AB6A85EDF59E712BA1D, taskId=task_1463533064464_0001_1_01_000000, scheduledTime=1463533064704, launchTime=1463533064704
2016-05-18 00:57:44,704 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]
2016-05-18 00:57:44,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533064464_0001_00_000002, launchTime=1463533064705
2016-05-18 00:57:44,706 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533064464_0001_00_000002
2016-05-18 00:57:44,707 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533064464_0001_1_01_000000_0] started. Is using containerId: [container_1463533064464_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:44,707 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CF72614CA9B24AB6A85EDF59E712BA1D, taskAttemptId=attempt_1463533064464_0001_1_01_000000_0, startTime=1463533064707, containerId=container_1463533064464_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:44,707 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533064464_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533064464_0001_1_01_000000_0
2016-05-18 00:57:44 Starting to run new task attempt: attempt_1463533064464_0001_1_01_000000_0
2016-05-18 00:57:44,708 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533064464_0001_1_01_000000_0
2016-05-18 00:57:44,708 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,708 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:44,708 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD] every/(1/1) ...lyplatform/0_ARGS_RESULTS, VertexName: CF72614CA9B24AB6A85EDF59E712BA1D, VertexParallelism: 1, TaskAttemptID:attempt_1463533064464_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BC757255BE9E4BB78D5C53E9FF951358, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=222E6B4F247A48B2B17F679616223290, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:44,708 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:44,709 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533064464_0001_1_01_000000_0
2016-05-18 00:57:44,709 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533064464_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]
2016-05-18 00:57:44,709 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533064464_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:44,712 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:44,712 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533064464_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:44,712 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533064464_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:44,713 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533064464_0001_00_000001, Exiting
2016-05-18 00:57:44,713 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533064464_0001_00_000001
2016-05-18 00:57:44,713 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533064464_0001_00_000001
2016-05-18 00:57:44,715 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:44,717 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CF72614CA9B24AB6A85EDF59E712BA1D, ordinal: 1
2016-05-18 00:57:44,717 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF72614CA9B24AB6A85EDF59E712BA1D, mem on start (mb), free: 169, total: 365, max: 672
2016-05-18 00:57:44,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:44,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:44,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:44,718 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533064464_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:44,718 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533064464_0001_00_000001 completed successfully
2016-05-18 00:57:44,718 INFO output.MROutput (MROutput.java:initialize(421)) - 222E6B4F247A48B2B17F679616223290: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:44,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:44,718 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[222E6B4F247A48B2B17F679616223290:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [BC757255BE9E4BB78D5C53E9FF951358:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:44,719 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:44,719 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:44,719 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:44,719 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[222E6B4F247A48B2B17F679616223290:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [BC757255BE9E4BB78D5C53E9FF951358:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:44,719 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:44,719 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BC757255BE9E4BB78D5C53E9FF951358 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:44,719 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BC757255BE9E4BB78D5C53E9FF951358: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:44,719 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:44,720 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:44,720 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BC757255BE9E4BB78D5C53E9FF951358: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:44,721 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BC757255BE9E4BB78D5C53E9FF951358 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:44,722 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BC757255BE9E4BB78D5C53E9FF951358
2016-05-18 00:57:44,722 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:44,723 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533064464_0001_1_01_000000_0
2016-05-18 00:57:44,724 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BC757255BE9E4BB78D5C53E9FF951358: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:44,725 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:44,725 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533064464_0001_1_00_000000_0_10213} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533064725, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:44,726 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BC757255BE9E4BB78D5C53E9FF951358
2016-05-18 00:57:44,726 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: F6BEDA400CC642ABA666C1060A266CB9
2016-05-18 00:57:44,726 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_RESULTS"], id: 222E6B4F247A48B2B17F679616223290
2016-05-18 00:57:44,726 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:44,726 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_RESULTS"]
2016-05-18 00:57:44,727 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BC757255BE9E4BB78D5C53E9FF951358, wasInterrupted=false
2016-05-18 00:57:44,727 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] F6BEDA400CC642ABA666C1060A266CB9, for 1 inputs
2016-05-18 00:57:44,727 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:44,728 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:44,729 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:44,729 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001_wd/localmode-local-dir/output/attempt_1463533064464_0001_1_00_000000_0_10213/file.out, len=181, isLocal=true
2016-05-18 00:57:44,729 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:44,729 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:44,729 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:44,730 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:44,730 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BC757255BE9E4BB78D5C53E9FF951358
2016-05-18 00:57:44,730 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BC757255BE9E4BB78D5C53E9FF951358: Shuffle Runner thread complete
2016-05-18 00:57:44,730 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CF72614CA9B24AB6A85EDF59E712BA1D, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:44,731 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:44,731 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BC757255BE9E4BB78D5C53E9FF951358: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:44,736 INFO output.MROutput (MROutput.java:close(526)) - 222E6B4F247A48B2B17F679616223290 closed
2016-05-18 00:57:44,736 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533064464_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:44,737 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533064464_0001_1_01_000000_0
2016-05-18 00:57:44,737 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CF72614CA9B24AB6A85EDF59E712BA1D, mem on close (mb), free: 165, total: 365, max: 672
2016-05-18 00:57:44,737 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BC757255BE9E4BB78D5C53E9FF951358
2016-05-18 00:57:44,737 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BC757255BE9E4BB78D5C53E9FF951358: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:44,737 INFO output.MROutput (MROutput.java:close(526)) - 222E6B4F247A48B2B17F679616223290 closed
2016-05-18 00:57:44,738 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:44 Completed running task attempt: attempt_1463533064464_0001_1_01_000000_0
2016-05-18 00:57:44,738 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533064464_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:44,738 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533064464_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:44,739 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533064464_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533064730, Process_Duration=7, Process_End_Time=1463533064737, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:44,739 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533064464_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:44,739 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:44,739 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:44,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CF72614CA9B24AB6A85EDF59E712BA1D, taskAttemptId=attempt_1463533064464_0001_1_01_000000_0, creationTime=1463533064704, allocationTime=1463533064705, startTime=1463533064707, finishTime=1463533064739, timeTaken=32, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:44,739 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,740 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533064464_0001_1_01_000000_0
2016-05-18 00:57:44,740 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533064464_0001_00_000002
2016-05-18 00:57:44,740 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:TASK_FINISHED]: vertexName=CF72614CA9B24AB6A85EDF59E712BA1D, taskId=task_1463533064464_0001_1_01_000000, startTime=1463533064707, finishTime=1463533064740, timeTaken=33, status=SUCCEEDED, successfulAttemptID=attempt_1463533064464_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533064730, Process_Duration=7, Process_End_Time=1463533064737, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:44,740 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:44,740 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D]
2016-05-18 00:57:44,741 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:VERTEX_FINISHED]: vertexName=CF72614CA9B24AB6A85EDF59E712BA1D, vertexId=vertex_1463533064464_0001_1_01, initRequestedTime=1463533064633, initedTime=1463533064637, startRequestedTime=1463533064644, startedTime=1463533064644, finishTime=1463533064741, timeTaken=97, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533064730, Process_Duration=7, Process_End_Time=1463533064737, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533064707, firstTasksToStart=[ task_1463533064464_0001_1_01_000000 ], lastTaskFinishTime=1463533064739, lastTasksToFinish=[ task_1463533064464_0001_1_01_000000 ], minTaskDuration=32, maxTaskDuration=32, avgTaskDuration=32.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533064464_0001_1_01_000000 ], longestDurationTasks=[ task_1463533064464_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:44,741 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:44,741 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:44,741 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:44,742 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533064464_0001_1_00 [BC757255BE9E4BB78D5C53E9FF951358]
2016-05-18 00:57:44,742 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533064464_0001_1
2016-05-18 00:57:44,742 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533064464_0001_1
2016-05-18 00:57:44,742 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533064464_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:44,742 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 222E6B4F247A48B2B17F679616223290 for vertex: vertex_1463533064464_0001_1_01 [CF72614CA9B24AB6A85EDF59E712BA1D], outputName: 222E6B4F247A48B2B17F679616223290
2016-05-18 00:57:44,742 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533064464_0001_00_000002, stoppedTime=1463533064742, exitStatus=0
2016-05-18 00:57:44,742 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533064464_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:44,748 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:222E6B4F247A48B2B17F679616223290 of vertex/vertexGroup:CF72614CA9B24AB6A85EDF59E712BA1D isVertexGroupOutput:false
2016-05-18 00:57:44,748 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:44,749 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533064464_0001_1, queueSize=0
2016-05-18 00:57:44,749 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533064464_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533064464_0001_1, startTime=1463533064632, finishTime=1463533064748, timeTaken=116, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=230, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066129421, Process_Duration=9, Process_End_Time=2927066129430, Read_Duration=2, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:44,749 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533064464_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:44,749 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533064464_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:44 Completed Dag: dag_1463533064464_0001_1
2016-05-18 00:57:44,750 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533064464_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:44,750 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533064464_0001_1_post
2016-05-18 00:57:44,750 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:44,750 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:44,750 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:44,750 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD] every/(1/1) ...lyplatform/0_ARGS_RESULTS, with id=dag_1463533064464_0001_1
2016-05-18 00:57:44,750 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD] every/(1/1) ...lyplatform/0_ARGS_RESULTS, with id=dag_1463533064464_0001_1
2016-05-18 00:57:44,751 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533064464_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:44,751 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533064464_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:44,751 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533064464_0001_00_000002, Exiting
2016-05-18 00:57:44,751 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533064464_0001_00_000002
2016-05-18 00:57:44,751 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533064464_0001_00_000002
2016-05-18 00:57:44,751 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533064464_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:44,751 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533064464_0001_00_000002 completed successfully
2016-05-18 00:57:44,761 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/0_ARGS_RESULTS, applicationId=application_1463533064464_0001
2016-05-18 00:57:44,764 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:44,764 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:44,764 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:44,764 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:44,764 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:44,764 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:44,765 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:44,765 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:44,765 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:44,765 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:44,766 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:44,766 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:44,766 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:44,767 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:44,770 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E0C38E389D534CA3852081D6A4C69E44/E785E053E57E4420A54CA99646C920DD/.tez/application_1463533064464_0001
2016-05-18 00:57:44,770 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_RESULTS/_temporary
2016-05-18 00:57:44,771 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.329
2016-05-18 00:57:44,835 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_ARGS_RESULTS
2016-05-18 00:57:44,857 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:44,857 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:44,884 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.002
2016-05-18 00:57:44,886 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.004
2016-05-18 00:57:44,886 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:44,887 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:44,887 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_ARGS_ALL
2016-05-18 00:57:44,908 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:44,908 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:44,942 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.009
2016-05-18 00:57:44,951 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-18 00:57:44,951 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:44,951 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:44,951 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:44,970 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:44,975 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_REPLACE
2016-05-18 00:57:45,028 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:45,029 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:45,029 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_REPLACE"]
2016-05-18 00:57:45,029 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:45,029 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:45,029 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:45,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/0_ARGS_REPLACE
2016-05-18 00:57:45,030 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:45,042 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:45,046 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:45,046 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:45,046 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:45,055 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001 doesn't exist and is created
2016-05-18 00:57:45,069 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:45,069 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:45,069 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001_wd
2016-05-18 00:57:45,078 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533065046_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:45,079 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:45,079 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:45,080 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:45,080 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:45,080 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:45,080 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:45,081 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:45,082 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:45,082 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:45,082 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:45,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:45,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:45,083 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:45,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:45,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:45,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:45,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:45,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:45,083 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:45,085 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:45,086 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:45,086 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:45,086 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001_wd/localmode-log-dir
2016-05-18 00:57:45,086 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001_wd/localmode-log-dir/history.txt.appattempt_1463533065046_0001_000000, maxErrors=10
2016-05-18 00:57:45,087 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:45,087 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533065046_0001_000000, appSubmitTime=1463533065078, launchTime=1463533065078
2016-05-18 00:57:45,089 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:45,092 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:51932
2016-05-18 00:57:45,094 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533065046
2016-05-18 00:57:45,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533065046_0001_000000, startTime=1463533065096
2016-05-18 00:57:45,096 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:45,169 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:45,169 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:45,169 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/0_ARGS_REPLACE, applicationId=application_1463533065046_0001, dagName=[9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE] every/(1/1) ...lyplatform/0_ARGS_REPLACE
2016-05-18 00:57:45,184 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE] every/(1/1) ...lyplatform/0_ARGS_REPLACE
2016-05-18 00:57:45,184 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:45,185 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533065046_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001_wd/localmode-log-dir/dag_1463533065046_0001_1.dot
2016-05-18 00:57:45,185 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001_wd/localmode-log-dir/dag_1463533065046_0001_1-tez-dag.pb.txt
2016-05-18 00:57:45,192 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533065046_0001_1
2016-05-18 00:57:45,192 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,192 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE] every/(1/1) ...lyplatform/0_ARGS_REPLACE
2016-05-18 00:57:45 Running Dag: dag_1463533065046_0001_1
2016-05-18 00:57:45,202 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533065046_0001_1, submitTime=1463533065184
2016-05-18 00:57:45,203 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001/tez-conf.pb" } size: 36913 timestamp: 1463533065000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:45,203 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,203 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:45,203 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001/tez-conf.pb" } size: 36913 timestamp: 1463533065000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:45,203 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:45,203 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]
2016-05-18 00:57:45,204 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:45,204 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:45,204 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:45,204 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533065046_0001_1, initTime=1463533065204
2016-05-18 00:57:45,205 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533065046_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:45,205 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:45,207 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:DAG_STARTED]: dagID=dag_1463533065046_0001_1, startTime=1463533065207
2016-05-18 00:57:45,207 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533065046_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:45,207 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: EDEED7D143994DB88BE946B87C8D9C80 : {67AF755FE07E4BD68157E6BB848B709C={InputName=67AF755FE07E4BD68157E6BB848B709C}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:45,207 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 67AF755FE07E4BD68157E6BB848B709C, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:45,207 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,207 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,208 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,208 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,208 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:45,208 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]
2016-05-18 00:57:45,208 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 67AF755FE07E4BD68157E6BB848B709C on vertex vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,210 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/0_ARGS_REPLACE, applicationId=application_1463533065046_0001, dagName=[9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE] every/(1/1) ...lyplatform/0_ARGS_REPLACE
2016-05-18 00:57:45,211 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:45,211 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]
2016-05-18 00:57:45,211 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]
2016-05-18 00:57:45,212 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533065046_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:45,212 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4], numAdditionalOutputs=1
2016-05-18 00:57:45,212 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=C99D3A7F99AE4254B9656D45C24C9879, vertex=vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:45,213 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 67AF755FE07E4BD68157E6BB848B709C asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:45,214 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for ED0FB2A24F6647E9B7C6E509DE20C3E4:C99D3A7F99AE4254B9656D45C24C9879 using oldmapred API
2016-05-18 00:57:45,214 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=C99D3A7F99AE4254B9656D45C24C9879, vertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:45,215 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4, vertexId=vertex_1463533065046_0001_1_01, initRequestedTime=1463533065208, initedTime=1463533065212, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:45,215 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:45,216 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:45,216 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:45,216 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:45,216 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:45,217 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 67AF755FE07E4BD68157E6BB848B709C on vertex vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,217 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:45,218 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {67AF755FE07E4BD68157E6BB848B709C=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:45,218 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] parallelism set to 1
2016-05-18 00:57:45,218 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:45,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EDEED7D143994DB88BE946B87C8D9C80, vertexId=vertex_1463533065046_0001_1_00, initRequestedTime=1463533065207, initedTime=1463533065219, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:45,219 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] with distanceFromRoot: 0
2016-05-18 00:57:45,219 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:45,220 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533065046_0001_1_00, startRequestedTime=1463533065208, startedTime=1463533065220
2016-05-18 00:57:45,220 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533065046_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:45,220 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:45,221 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533065046_0001_1_00 for vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4] numStartedSources: 1 numSources: 1
2016-05-18 00:57:45,221 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4] with distanceFromRoot: 1
2016-05-18 00:57:45,221 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in EDEED7D143994DB88BE946B87C8D9C80
2016-05-18 00:57:45,221 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,222 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533065046_0001_1_01, startRequestedTime=1463533065221, startedTime=1463533065221
2016-05-18 00:57:45,222 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:45,222 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: ED0FB2A24F6647E9B7C6E509DE20C3E4 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:45,223 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: EDEED7D143994DB88BE946B87C8D9C80 in vertex: ED0FB2A24F6647E9B7C6E509DE20C3E4 numBipartiteSourceTasks: 1
2016-05-18 00:57:45,223 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:TASK_STARTED]: vertexName=EDEED7D143994DB88BE946B87C8D9C80, taskId=task_1463533065046_0001_1_00_000000, scheduledTime=1463533065222, launchTime=1463533065222
2016-05-18 00:57:45,224 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,224 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:45,226 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533065046_0001_00_000001, launchTime=1463533065226
2016-05-18 00:57:45,226 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533065046_0001_00_000001
2016-05-18 00:57:45,227 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533065046_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533065046_0001_1_00_000000_0
2016-05-18 00:57:45,227 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533065046_0001_1_00_000000_0] started. Is using containerId: [container_1463533065046_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:45,227 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EDEED7D143994DB88BE946B87C8D9C80, taskAttemptId=attempt_1463533065046_0001_1_00_000000_0, startTime=1463533065227, containerId=container_1463533065046_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:45 Starting to run new task attempt: attempt_1463533065046_0001_1_00_000000_0
2016-05-18 00:57:45,228 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533065046_0001_1_00_000000_0
2016-05-18 00:57:45,228 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,229 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:45,229 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE] every/(1/1) ...lyplatform/0_ARGS_REPLACE, VertexName: EDEED7D143994DB88BE946B87C8D9C80, VertexParallelism: 1, TaskAttemptID:attempt_1463533065046_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=67AF755FE07E4BD68157E6BB848B709C, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:45,229 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:45,230 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533065046_0001_1_00_000000_0
2016-05-18 00:57:45,230 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533065046_0001, with dag id: dag_1463533065046_0001_1
2016-05-18 00:57:45,230 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533065046_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:45,231 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533065046_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:45,232 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:45,237 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EDEED7D143994DB88BE946B87C8D9C80, ordinal: 0
2016-05-18 00:57:45,237 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EDEED7D143994DB88BE946B87C8D9C80, mem on start (mb), free: 251, total: 364, max: 672
2016-05-18 00:57:45,237 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:45,237 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:45,237 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:45,238 INFO input.MRInput (MRInput.java:initialize(440)) - 67AF755FE07E4BD68157E6BB848B709C using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:45,238 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 67AF755FE07E4BD68157E6BB848B709C
2016-05-18 00:57:45,238 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:45,238 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[67AF755FE07E4BD68157E6BB848B709C:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [ED0FB2A24F6647E9B7C6E509DE20C3E4:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:45,238 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:45,239 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:45,239 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:45,239 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[67AF755FE07E4BD68157E6BB848B709C:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [ED0FB2A24F6647E9B7C6E509DE20C3E4:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:45,239 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:45,239 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 67AF755FE07E4BD68157E6BB848B709C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:45,239 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:45,240 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:45,240 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 67AF755FE07E4BD68157E6BB848B709C
2016-05-18 00:57:45,240 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:45,240 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533065046_0001_1_00_000000_0
2016-05-18 00:57:45,242 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:45,243 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 67AF755FE07E4BD68157E6BB848B709C initialized RecordReader from event
2016-05-18 00:57:45,243 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:45,244 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 67AF755FE07E4BD68157E6BB848B709C
2016-05-18 00:57:45,244 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: A0C3004450914C88AFB855CA844E46CC
2016-05-18 00:57:45,244 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] A0C3004450914C88AFB855CA844E46CC
2016-05-18 00:57:45,245 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:45,247 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:45,247 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - ED0FB2A24F6647E9B7C6E509DE20C3E4 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@325a4199, comparator=cascading.tuple.hadoop.util.TupleComparator@4388bda2, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:45,247 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for ED0FB2A24F6647E9B7C6E509DE20C3E4: , UsingHashComparator=false
2016-05-18 00:57:45,260 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:45,260 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:45,260 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for ED0FB2A24F6647E9B7C6E509DE20C3E4: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:45,260 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - ED0FB2A24F6647E9B7C6E509DE20C3E4: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:45,260 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:45,260 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EDEED7D143994DB88BE946B87C8D9C80, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:45,263 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533065046_0001_1_00_000000_0
2016-05-18 00:57:45,263 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EDEED7D143994DB88BE946B87C8D9C80, mem on close (mb), free: 150, total: 364, max: 672
2016-05-18 00:57:45,263 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - ED0FB2A24F6647E9B7C6E509DE20C3E4: Starting flush of map output
2016-05-18 00:57:45,263 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - ED0FB2A24F6647E9B7C6E509DE20C3E4: Span0.length = 20, perItem = 6
2016-05-18 00:57:45,264 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - ED0FB2A24F6647E9B7C6E509DE20C3E4: done sorting span=0, length=20, time=1
2016-05-18 00:57:45,264 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - ED0FB2A24F6647E9B7C6E509DE20C3E4: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:45,265 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - ED0FB2A24F6647E9B7C6E509DE20C3E4: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001_wd/localmode-local-dir/output/attempt_1463533065046_0001_1_00_000000_0_10219_0/file.out
2016-05-18 00:57:45,269 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:45 Completed running task attempt: attempt_1463533065046_0001_1_00_000000_0
2016-05-18 00:57:45,269 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533065046_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:45,269 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533065046_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:45,270 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533065046_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533065260, Process_Duration=3, Process_End_Time=1463533065263, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:45,270 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533065046_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:45,270 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:45,270 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EDEED7D143994DB88BE946B87C8D9C80, taskAttemptId=attempt_1463533065046_0001_1_00_000000_0, creationTime=1463533065223, allocationTime=1463533065225, startTime=1463533065227, finishTime=1463533065270, timeTaken=43, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:45,271 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:45,271 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,271 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533065046_0001_00_000001
2016-05-18 00:57:45,271 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533065046_0001_1_00_000000_0
2016-05-18 00:57:45,271 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:TASK_FINISHED]: vertexName=EDEED7D143994DB88BE946B87C8D9C80, taskId=task_1463533065046_0001_1_00_000000, startTime=1463533065227, finishTime=1463533065271, timeTaken=44, status=SUCCEEDED, successfulAttemptID=attempt_1463533065046_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533065260, Process_Duration=3, Process_End_Time=1463533065263, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:45,271 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:45,271 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,272 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:VERTEX_FINISHED]: vertexName=EDEED7D143994DB88BE946B87C8D9C80, vertexId=vertex_1463533065046_0001_1_00, initRequestedTime=1463533065207, initedTime=1463533065219, startRequestedTime=1463533065208, startedTime=1463533065220, finishTime=1463533065271, timeTaken=51, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533065260, Process_Duration=3, Process_End_Time=1463533065263, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533065227, firstTasksToStart=[ task_1463533065046_0001_1_00_000000 ], lastTaskFinishTime=1463533065270, lastTasksToFinish=[ task_1463533065046_0001_1_00_000000 ], minTaskDuration=43, maxTaskDuration=43, avgTaskDuration=43.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533065046_0001_1_00_000000 ], longestDurationTasks=[ task_1463533065046_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:45,272 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:45,272 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4] attempt: attempt_1463533065046_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:45,272 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:45,272 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:45,272 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: ED0FB2A24F6647E9B7C6E509DE20C3E4
2016-05-18 00:57:45,273 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=EDEED7D143994DB88BE946B87C8D9C80, destinationVertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:45,273 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]
2016-05-18 00:57:45,273 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533065046_0001_00_000001, stoppedTime=1463533065273, exitStatus=0
2016-05-18 00:57:45,273 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:TASK_STARTED]: vertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4, taskId=task_1463533065046_0001_1_01_000000, scheduledTime=1463533065273, launchTime=1463533065273
2016-05-18 00:57:45,273 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]
2016-05-18 00:57:45,273 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533065046_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:45,275 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533065046_0001_00_000002, launchTime=1463533065275
2016-05-18 00:57:45,275 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533065046_0001_00_000002
2016-05-18 00:57:45,276 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533065046_0001_1_01_000000_0] started. Is using containerId: [container_1463533065046_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:45,276 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4, taskAttemptId=attempt_1463533065046_0001_1_01_000000_0, startTime=1463533065276, containerId=container_1463533065046_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:45,276 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533065046_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533065046_0001_1_01_000000_0
2016-05-18 00:57:45 Starting to run new task attempt: attempt_1463533065046_0001_1_01_000000_0
2016-05-18 00:57:45,277 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533065046_0001_1_01_000000_0
2016-05-18 00:57:45,277 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,277 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:45,277 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE] every/(1/1) ...lyplatform/0_ARGS_REPLACE, VertexName: ED0FB2A24F6647E9B7C6E509DE20C3E4, VertexParallelism: 1, TaskAttemptID:attempt_1463533065046_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EDEED7D143994DB88BE946B87C8D9C80, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C99D3A7F99AE4254B9656D45C24C9879, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:45,278 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:45,278 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533065046_0001_1_01_000000_0
2016-05-18 00:57:45,280 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533065046_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]
2016-05-18 00:57:45,281 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533065046_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:45,281 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533065046_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:45,281 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533065046_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:45,282 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533065046_0001_00_000001, Exiting
2016-05-18 00:57:45,282 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533065046_0001_00_000001
2016-05-18 00:57:45,282 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533065046_0001_00_000001
2016-05-18 00:57:45,282 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:45,283 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533065046_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:45,283 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533065046_0001_00_000001 completed successfully
2016-05-18 00:57:45,285 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:45,287 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ED0FB2A24F6647E9B7C6E509DE20C3E4, ordinal: 1
2016-05-18 00:57:45,287 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ED0FB2A24F6647E9B7C6E509DE20C3E4, mem on start (mb), free: 145, total: 364, max: 672
2016-05-18 00:57:45,287 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:45,287 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:45,287 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:45,288 INFO output.MROutput (MROutput.java:initialize(421)) - C99D3A7F99AE4254B9656D45C24C9879: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:45,288 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:45,288 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[C99D3A7F99AE4254B9656D45C24C9879:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [EDEED7D143994DB88BE946B87C8D9C80:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:45,288 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:45,289 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:45,289 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:45,289 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[C99D3A7F99AE4254B9656D45C24C9879:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [EDEED7D143994DB88BE946B87C8D9C80:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:45,289 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:45,289 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: EDEED7D143994DB88BE946B87C8D9C80 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:45,289 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - EDEED7D143994DB88BE946B87C8D9C80: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:45,289 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - EDEED7D143994DB88BE946B87C8D9C80: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:45,290 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: EDEED7D143994DB88BE946B87C8D9C80 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:45,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: EDEED7D143994DB88BE946B87C8D9C80
2016-05-18 00:57:45,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:45,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:45,295 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:45,295 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533065046_0001_1_01_000000_0
2016-05-18 00:57:45,296 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EDEED7D143994DB88BE946B87C8D9C80: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:45,297 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533065046_0001_1_00_000000_0_10219} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533065297, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:45,297 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : EDEED7D143994DB88BE946B87C8D9C80
2016-05-18 00:57:45,297 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:45,297 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: EDEED7D143994DB88BE946B87C8D9C80, wasInterrupted=false
2016-05-18 00:57:45,298 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:45,298 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:45,299 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:45,299 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001_wd/localmode-local-dir/output/attempt_1463533065046_0001_1_00_000000_0_10219/file.out, len=181, isLocal=true
2016-05-18 00:57:45,299 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:45,299 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:45,299 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:45,300 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:45,300 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : EDEED7D143994DB88BE946B87C8D9C80
2016-05-18 00:57:45,300 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - EDEED7D143994DB88BE946B87C8D9C80: Shuffle Runner thread complete
2016-05-18 00:57:45,300 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:45,301 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: A0C3004450914C88AFB855CA844E46CC
2016-05-18 00:57:45,301 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_REPLACE"], id: C99D3A7F99AE4254B9656D45C24C9879
2016-05-18 00:57:45,301 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_REPLACE"]
2016-05-18 00:57:45,301 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] A0C3004450914C88AFB855CA844E46CC, for 1 inputs
2016-05-18 00:57:45,302 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ED0FB2A24F6647E9B7C6E509DE20C3E4, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:45,303 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:45,303 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - EDEED7D143994DB88BE946B87C8D9C80: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:45,308 INFO output.MROutput (MROutput.java:close(526)) - C99D3A7F99AE4254B9656D45C24C9879 closed
2016-05-18 00:57:45,309 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533065046_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:45,309 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533065046_0001_1_01_000000_0
2016-05-18 00:57:45,309 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ED0FB2A24F6647E9B7C6E509DE20C3E4, mem on close (mb), free: 141, total: 364, max: 672
2016-05-18 00:57:45,309 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: EDEED7D143994DB88BE946B87C8D9C80
2016-05-18 00:57:45,309 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EDEED7D143994DB88BE946B87C8D9C80: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:45,310 INFO output.MROutput (MROutput.java:close(526)) - C99D3A7F99AE4254B9656D45C24C9879 closed
2016-05-18 00:57:45,310 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:45 Completed running task attempt: attempt_1463533065046_0001_1_01_000000_0
2016-05-18 00:57:45,310 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533065046_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:45,310 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533065046_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:45,311 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533065046_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533065302, Process_Duration=7, Process_End_Time=1463533065309, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:45,311 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533065046_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:45,311 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:45,311 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4, taskAttemptId=attempt_1463533065046_0001_1_01_000000_0, creationTime=1463533065273, allocationTime=1463533065274, startTime=1463533065276, finishTime=1463533065311, timeTaken=35, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:45,311 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:45,312 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,312 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533065046_0001_1_01_000000_0
2016-05-18 00:57:45,312 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533065046_0001_00_000002
2016-05-18 00:57:45,312 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:TASK_FINISHED]: vertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4, taskId=task_1463533065046_0001_1_01_000000, startTime=1463533065276, finishTime=1463533065312, timeTaken=36, status=SUCCEEDED, successfulAttemptID=attempt_1463533065046_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533065302, Process_Duration=7, Process_End_Time=1463533065309, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:45,312 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:45,312 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4]
2016-05-18 00:57:45,313 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:VERTEX_FINISHED]: vertexName=ED0FB2A24F6647E9B7C6E509DE20C3E4, vertexId=vertex_1463533065046_0001_1_01, initRequestedTime=1463533065208, initedTime=1463533065212, startRequestedTime=1463533065221, startedTime=1463533065221, finishTime=1463533065312, timeTaken=91, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533065302, Process_Duration=7, Process_End_Time=1463533065309, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533065276, firstTasksToStart=[ task_1463533065046_0001_1_01_000000 ], lastTaskFinishTime=1463533065311, lastTasksToFinish=[ task_1463533065046_0001_1_01_000000 ], minTaskDuration=35, maxTaskDuration=35, avgTaskDuration=35.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533065046_0001_1_01_000000 ], longestDurationTasks=[ task_1463533065046_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:45,313 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:45,313 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:45,313 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:45,313 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533065046_0001_1_00 [EDEED7D143994DB88BE946B87C8D9C80]
2016-05-18 00:57:45,313 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533065046_0001_1
2016-05-18 00:57:45,313 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533065046_0001_1
2016-05-18 00:57:45,313 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533065046_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:45,313 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: C99D3A7F99AE4254B9656D45C24C9879 for vertex: vertex_1463533065046_0001_1_01 [ED0FB2A24F6647E9B7C6E509DE20C3E4], outputName: C99D3A7F99AE4254B9656D45C24C9879
2016-05-18 00:57:45,314 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533065046_0001_00_000002, stoppedTime=1463533065314, exitStatus=0
2016-05-18 00:57:45,314 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533065046_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:45,320 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:C99D3A7F99AE4254B9656D45C24C9879 of vertex/vertexGroup:ED0FB2A24F6647E9B7C6E509DE20C3E4 isVertexGroupOutput:false
2016-05-18 00:57:45,320 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:45,322 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533065046_0001_1, queueSize=0
2016-05-18 00:57:45,322 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065046_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533065046_0001_1, startTime=1463533065205, finishTime=1463533065321, timeTaken=116, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=200, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066130562, Process_Duration=10, Process_End_Time=2927066130572, Read_Duration=2, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:45,322 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533065046_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:45,322 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533065046_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:45 Completed Dag: dag_1463533065046_0001_1
2016-05-18 00:57:45,322 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533065046_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:45,322 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533065046_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:45,323 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533065046_0001_1_post
2016-05-18 00:57:45,323 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,323 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:45,323 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533065046_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:45,323 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:45,323 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533065046_0001_00_000002, Exiting
2016-05-18 00:57:45,323 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE] every/(1/1) ...lyplatform/0_ARGS_REPLACE, with id=dag_1463533065046_0001_1
2016-05-18 00:57:45,323 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533065046_0001_00_000002
2016-05-18 00:57:45,323 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE] every/(1/1) ...lyplatform/0_ARGS_REPLACE, with id=dag_1463533065046_0001_1
2016-05-18 00:57:45,323 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533065046_0001_00_000002
2016-05-18 00:57:45,324 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533065046_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:45,324 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533065046_0001_00_000002 completed successfully
2016-05-18 00:57:45,334 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/0_ARGS_REPLACE, applicationId=application_1463533065046_0001
2016-05-18 00:57:45,336 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:45,336 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:45,336 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:45,336 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:45,336 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:45,337 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:45,337 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:45,337 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:45,337 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:45,337 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:45,338 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:45,338 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:45,338 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:45,339 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:45,343 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9E49CAD8CB5B4214825A6302DD8909C4/ED873E3C574C430C906CF2D1984133EE/.tez/application_1463533065046_0001
2016-05-18 00:57:45,343 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_REPLACE/_temporary
2016-05-18 00:57:45,344 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.315
2016-05-18 00:57:45,413 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_ARGS_REPLACE
2016-05-18 00:57:45,435 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:45,436 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:45,466 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:45,472 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 00:57:45,472 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:45,473 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:45,473 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:45,488 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:45,493 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_SWAP
2016-05-18 00:57:45,547 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:45,547 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:45,548 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_SWAP"]
2016-05-18 00:57:45,548 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:45,548 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:45,548 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:45,549 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...emblyplatform/0_ARGS_SWAP
2016-05-18 00:57:45,549 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:45,562 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:45,566 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:45,566 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:45,566 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:45,575 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001 doesn't exist and is created
2016-05-18 00:57:45,589 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:45,589 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:45,589 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001_wd
2016-05-18 00:57:45,599 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533065566_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:45,599 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:45,600 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:45,601 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:45,601 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:45,601 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:45,602 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:45,602 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:45,604 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:45,604 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:45,604 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:45,604 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:45,604 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:45,605 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:45,605 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:45,605 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:45,605 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:45,606 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:45,606 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:45,606 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:45,607 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:45,608 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:45,608 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:45,609 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001_wd/localmode-log-dir
2016-05-18 00:57:45,609 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001_wd/localmode-log-dir/history.txt.appattempt_1463533065566_0001_000000, maxErrors=10
2016-05-18 00:57:45,609 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:45,609 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533065566_0001_000000, appSubmitTime=1463533065599, launchTime=1463533065599
2016-05-18 00:57:45,612 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:45,615 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:44731
2016-05-18 00:57:45,616 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533065566
2016-05-18 00:57:45,617 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533065566_0001_000000, startTime=1463533065617
2016-05-18 00:57:45,617 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:45,689 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:45,689 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:45,689 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...emblyplatform/0_ARGS_SWAP, applicationId=application_1463533065566_0001, dagName=[F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5] every/(1/1) ...emblyplatform/0_ARGS_SWAP
2016-05-18 00:57:45,704 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5] every/(1/1) ...emblyplatform/0_ARGS_SWAP
2016-05-18 00:57:45,704 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:45,705 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533065566_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001_wd/localmode-log-dir/dag_1463533065566_0001_1.dot
2016-05-18 00:57:45,705 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001_wd/localmode-log-dir/dag_1463533065566_0001_1-tez-dag.pb.txt
2016-05-18 00:57:45,712 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533065566_0001_1
2016-05-18 00:57:45,712 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,712 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5] every/(1/1) ...emblyplatform/0_ARGS_SWAP
2016-05-18 00:57:45 Running Dag: dag_1463533065566_0001_1
2016-05-18 00:57:45,720 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533065566_0001_1, submitTime=1463533065704
2016-05-18 00:57:45,721 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001/tez-conf.pb" } size: 36913 timestamp: 1463533065000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:45,722 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,722 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:45,723 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001/tez-conf.pb" } size: 36913 timestamp: 1463533065000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:45,724 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]
2016-05-18 00:57:45,724 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:45,724 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:45,725 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533065566_0001_1, initTime=1463533065724
2016-05-18 00:57:45,725 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533065566_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:45,726 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:45,726 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:DAG_STARTED]: dagID=dag_1463533065566_0001_1, startTime=1463533065726
2016-05-18 00:57:45,726 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533065566_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:45,726 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E92CD571D4734E478DD368290DA28EA6 : {A9E9CAD6A708420388C70ED16584D029={InputName=A9E9CAD6A708420388C70ED16584D029}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:45,727 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: A9E9CAD6A708420388C70ED16584D029, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:45,727 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,727 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,727 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...emblyplatform/0_ARGS_SWAP, applicationId=application_1463533065566_0001, dagName=[F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5] every/(1/1) ...emblyplatform/0_ARGS_SWAP
2016-05-18 00:57:45,727 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,727 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,727 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:45,727 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: A9E9CAD6A708420388C70ED16584D029 on vertex vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,727 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]
2016-05-18 00:57:45,730 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:45,730 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]
2016-05-18 00:57:45,730 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]
2016-05-18 00:57:45,731 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533065566_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:45,731 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31], numAdditionalOutputs=1
2016-05-18 00:57:45,731 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=F4C9A4836B984E45821FD5621473B27B, vertex=vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:45,733 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for B04769583C1D4B62B8EC726EB6BABD31:F4C9A4836B984E45821FD5621473B27B using oldmapred API
2016-05-18 00:57:45,733 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=F4C9A4836B984E45821FD5621473B27B, vertexName=B04769583C1D4B62B8EC726EB6BABD31, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:45,733 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input A9E9CAD6A708420388C70ED16584D029 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:45,734 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B04769583C1D4B62B8EC726EB6BABD31, vertexId=vertex_1463533065566_0001_1_01, initRequestedTime=1463533065727, initedTime=1463533065731, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:45,734 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:45,735 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:45,735 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:45,735 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:45,735 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:45,736 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: A9E9CAD6A708420388C70ED16584D029 on vertex vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,737 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:45,737 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {A9E9CAD6A708420388C70ED16584D029=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:45,738 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] parallelism set to 1
2016-05-18 00:57:45,739 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:45,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E92CD571D4734E478DD368290DA28EA6, vertexId=vertex_1463533065566_0001_1_00, initRequestedTime=1463533065726, initedTime=1463533065739, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:45,739 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] with distanceFromRoot: 0
2016-05-18 00:57:45,739 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:45,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533065566_0001_1_00, startRequestedTime=1463533065727, startedTime=1463533065739
2016-05-18 00:57:45,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533065566_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:45,740 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:45,740 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533065566_0001_1_00 for vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31] numStartedSources: 1 numSources: 1
2016-05-18 00:57:45,740 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31] with distanceFromRoot: 1
2016-05-18 00:57:45,740 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533065566_0001_1_01, startRequestedTime=1463533065740, startedTime=1463533065740
2016-05-18 00:57:45,740 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:45,741 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E92CD571D4734E478DD368290DA28EA6
2016-05-18 00:57:45,741 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,741 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:TASK_STARTED]: vertexName=E92CD571D4734E478DD368290DA28EA6, taskId=task_1463533065566_0001_1_00_000000, scheduledTime=1463533065741, launchTime=1463533065741
2016-05-18 00:57:45,742 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,741 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: B04769583C1D4B62B8EC726EB6BABD31 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:45,743 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E92CD571D4734E478DD368290DA28EA6 in vertex: B04769583C1D4B62B8EC726EB6BABD31 numBipartiteSourceTasks: 1
2016-05-18 00:57:45,743 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:45,745 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533065566_0001_00_000001, launchTime=1463533065745
2016-05-18 00:57:45,745 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533065566_0001_00_000001
2016-05-18 00:57:45,746 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533065566_0001_1_00_000000_0] started. Is using containerId: [container_1463533065566_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:45,746 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E92CD571D4734E478DD368290DA28EA6, taskAttemptId=attempt_1463533065566_0001_1_00_000000_0, startTime=1463533065746, containerId=container_1463533065566_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:45,746 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533065566_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533065566_0001_1_00_000000_0
2016-05-18 00:57:45 Starting to run new task attempt: attempt_1463533065566_0001_1_00_000000_0
2016-05-18 00:57:45,747 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533065566_0001_1_00_000000_0
2016-05-18 00:57:45,747 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,748 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:45,748 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5] every/(1/1) ...emblyplatform/0_ARGS_SWAP, VertexName: E92CD571D4734E478DD368290DA28EA6, VertexParallelism: 1, TaskAttemptID:attempt_1463533065566_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A9E9CAD6A708420388C70ED16584D029, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B04769583C1D4B62B8EC726EB6BABD31, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:45,749 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:45,749 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533065566_0001_1_00_000000_0
2016-05-18 00:57:45,750 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533065566_0001, with dag id: dag_1463533065566_0001_1
2016-05-18 00:57:45,750 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533065566_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:45,750 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533065566_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:45,752 INFO input.MRInput (MRInput.java:initialize(440)) - A9E9CAD6A708420388C70ED16584D029 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:45,752 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:45,752 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: A9E9CAD6A708420388C70ED16584D029
2016-05-18 00:57:45,756 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E92CD571D4734E478DD368290DA28EA6, ordinal: 0
2016-05-18 00:57:45,756 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E92CD571D4734E478DD368290DA28EA6, mem on start (mb), free: 246, total: 365, max: 672
2016-05-18 00:57:45,756 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:45,757 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:45,757 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:45,759 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:45,759 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A9E9CAD6A708420388C70ED16584D029:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [B04769583C1D4B62B8EC726EB6BABD31:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:45,760 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:45,760 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:45,760 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:45,760 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A9E9CAD6A708420388C70ED16584D029:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [B04769583C1D4B62B8EC726EB6BABD31:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:45,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:45,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A9E9CAD6A708420388C70ED16584D029 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:45,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:45,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:45,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A9E9CAD6A708420388C70ED16584D029
2016-05-18 00:57:45,762 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:45,762 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533065566_0001_1_00_000000_0
2016-05-18 00:57:45,762 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:45,762 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:45,763 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:45,763 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - A9E9CAD6A708420388C70ED16584D029 initialized RecordReader from event
2016-05-18 00:57:45,767 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:45,767 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: A9E9CAD6A708420388C70ED16584D029
2016-05-18 00:57:45,768 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 5212C1AF5E224208A4C7021B526DD423
2016-05-18 00:57:45,768 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 5212C1AF5E224208A4C7021B526DD423
2016-05-18 00:57:45,770 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:45,772 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:45,772 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - B04769583C1D4B62B8EC726EB6BABD31 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@803be8a, comparator=cascading.tuple.hadoop.util.TupleComparator@2dd25a22, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:45,773 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for B04769583C1D4B62B8EC726EB6BABD31: , UsingHashComparator=false
2016-05-18 00:57:45,786 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:45,787 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:45,787 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for B04769583C1D4B62B8EC726EB6BABD31: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:45,787 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - B04769583C1D4B62B8EC726EB6BABD31: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:45,787 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:45,788 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E92CD571D4734E478DD368290DA28EA6, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:45,790 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533065566_0001_1_00_000000_0
2016-05-18 00:57:45,790 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E92CD571D4734E478DD368290DA28EA6, mem on close (mb), free: 143, total: 365, max: 672
2016-05-18 00:57:45,790 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - B04769583C1D4B62B8EC726EB6BABD31: Starting flush of map output
2016-05-18 00:57:45,790 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - B04769583C1D4B62B8EC726EB6BABD31: Span0.length = 20, perItem = 6
2016-05-18 00:57:45,791 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - B04769583C1D4B62B8EC726EB6BABD31: done sorting span=0, length=20, time=1
2016-05-18 00:57:45,791 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - B04769583C1D4B62B8EC726EB6BABD31: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:45,792 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - B04769583C1D4B62B8EC726EB6BABD31: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001_wd/localmode-local-dir/output/attempt_1463533065566_0001_1_00_000000_0_10225_0/file.out
2016-05-18 00:57:45,797 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:45 Completed running task attempt: attempt_1463533065566_0001_1_00_000000_0
2016-05-18 00:57:45,798 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533065566_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:45,798 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533065566_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:45,798 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533065566_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533065788, Process_Duration=2, Process_End_Time=1463533065790, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:45,799 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533065566_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:45,799 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:45,799 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:45,799 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,799 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E92CD571D4734E478DD368290DA28EA6, taskAttemptId=attempt_1463533065566_0001_1_00_000000_0, creationTime=1463533065742, allocationTime=1463533065744, startTime=1463533065746, finishTime=1463533065799, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:45,800 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533065566_0001_1_00_000000_0
2016-05-18 00:57:45,801 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:TASK_FINISHED]: vertexName=E92CD571D4734E478DD368290DA28EA6, taskId=task_1463533065566_0001_1_00_000000, startTime=1463533065746, finishTime=1463533065800, timeTaken=54, status=SUCCEEDED, successfulAttemptID=attempt_1463533065566_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533065788, Process_Duration=2, Process_End_Time=1463533065790, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:45,801 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:45,801 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,802 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:VERTEX_FINISHED]: vertexName=E92CD571D4734E478DD368290DA28EA6, vertexId=vertex_1463533065566_0001_1_00, initRequestedTime=1463533065726, initedTime=1463533065739, startRequestedTime=1463533065727, startedTime=1463533065739, finishTime=1463533065801, timeTaken=62, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533065788, Process_Duration=2, Process_End_Time=1463533065790, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533065746, firstTasksToStart=[ task_1463533065566_0001_1_00_000000 ], lastTaskFinishTime=1463533065799, lastTasksToFinish=[ task_1463533065566_0001_1_00_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533065566_0001_1_00_000000 ], longestDurationTasks=[ task_1463533065566_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:45,802 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533065566_0001_00_000001
2016-05-18 00:57:45,802 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:45,802 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31] attempt: attempt_1463533065566_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:45,802 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B04769583C1D4B62B8EC726EB6BABD31
2016-05-18 00:57:45,802 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E92CD571D4734E478DD368290DA28EA6, destinationVertexName=B04769583C1D4B62B8EC726EB6BABD31. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:45,802 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533065566_0001_00_000001, stoppedTime=1463533065802, exitStatus=0
2016-05-18 00:57:45,802 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]
2016-05-18 00:57:45,802 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:45,803 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:45,803 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533065566_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:45,803 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:TASK_STARTED]: vertexName=B04769583C1D4B62B8EC726EB6BABD31, taskId=task_1463533065566_0001_1_01_000000, scheduledTime=1463533065803, launchTime=1463533065803
2016-05-18 00:57:45,803 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]
2016-05-18 00:57:45,805 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533065566_0001_00_000002, launchTime=1463533065805
2016-05-18 00:57:45,805 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533065566_0001_00_000002
2016-05-18 00:57:45,806 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533065566_0001_1_01_000000_0] started. Is using containerId: [container_1463533065566_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:45,806 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B04769583C1D4B62B8EC726EB6BABD31, taskAttemptId=attempt_1463533065566_0001_1_01_000000_0, startTime=1463533065806, containerId=container_1463533065566_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:45,807 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533065566_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533065566_0001_1_01_000000_0
2016-05-18 00:57:45 Starting to run new task attempt: attempt_1463533065566_0001_1_01_000000_0
2016-05-18 00:57:45,807 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533065566_0001_1_01_000000_0
2016-05-18 00:57:45,807 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,808 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:45,808 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5] every/(1/1) ...emblyplatform/0_ARGS_SWAP, VertexName: B04769583C1D4B62B8EC726EB6BABD31, VertexParallelism: 1, TaskAttemptID:attempt_1463533065566_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E92CD571D4734E478DD368290DA28EA6, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=F4C9A4836B984E45821FD5621473B27B, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:45,808 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:45,810 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533065566_0001_1_01_000000_0
2016-05-18 00:57:45,811 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533065566_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]
2016-05-18 00:57:45,812 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533065566_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:45,812 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533065566_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:45,812 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533065566_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:45,814 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:45,815 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:45,816 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533065566_0001_00_000001, Exiting
2016-05-18 00:57:45,816 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533065566_0001_00_000001
2016-05-18 00:57:45,816 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533065566_0001_00_000001
2016-05-18 00:57:45,820 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B04769583C1D4B62B8EC726EB6BABD31, ordinal: 1
2016-05-18 00:57:45,820 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B04769583C1D4B62B8EC726EB6BABD31, mem on start (mb), free: 139, total: 365, max: 672
2016-05-18 00:57:45,820 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:45,820 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:45,820 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:45,821 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533065566_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:45,821 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533065566_0001_00_000001 completed successfully
2016-05-18 00:57:45,827 INFO output.MROutput (MROutput.java:initialize(421)) - F4C9A4836B984E45821FD5621473B27B: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:45,827 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:45,828 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[E92CD571D4734E478DD368290DA28EA6:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [F4C9A4836B984E45821FD5621473B27B:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:45,828 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:45,828 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:45,828 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:45,828 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[E92CD571D4734E478DD368290DA28EA6:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [F4C9A4836B984E45821FD5621473B27B:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:45,828 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:45,828 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E92CD571D4734E478DD368290DA28EA6 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:45,829 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E92CD571D4734E478DD368290DA28EA6: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:45,829 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E92CD571D4734E478DD368290DA28EA6: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:45,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:45,830 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E92CD571D4734E478DD368290DA28EA6 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:45,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:45,830 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E92CD571D4734E478DD368290DA28EA6
2016-05-18 00:57:45,832 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:45,832 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533065566_0001_1_01_000000_0
2016-05-18 00:57:45,833 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E92CD571D4734E478DD368290DA28EA6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:45,834 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533065566_0001_1_00_000000_0_10225} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533065834, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:45,834 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E92CD571D4734E478DD368290DA28EA6
2016-05-18 00:57:45,835 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:45,835 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E92CD571D4734E478DD368290DA28EA6, wasInterrupted=false
2016-05-18 00:57:45,835 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:45,836 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:45,836 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:45,838 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 5212C1AF5E224208A4C7021B526DD423
2016-05-18 00:57:45,838 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:45,838 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_SWAP"], id: F4C9A4836B984E45821FD5621473B27B
2016-05-18 00:57:45,838 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001_wd/localmode-local-dir/output/attempt_1463533065566_0001_1_00_000000_0_10225/file.out, len=181, isLocal=true
2016-05-18 00:57:45,838 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_SWAP"]
2016-05-18 00:57:45,838 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:45,838 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 5212C1AF5E224208A4C7021B526DD423, for 1 inputs
2016-05-18 00:57:45,838 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:45,838 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:45,839 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:45,839 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E92CD571D4734E478DD368290DA28EA6
2016-05-18 00:57:45,839 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B04769583C1D4B62B8EC726EB6BABD31, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:57:45,839 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E92CD571D4734E478DD368290DA28EA6: Shuffle Runner thread complete
2016-05-18 00:57:45,840 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:45,840 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E92CD571D4734E478DD368290DA28EA6: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:45,846 INFO output.MROutput (MROutput.java:close(526)) - F4C9A4836B984E45821FD5621473B27B closed
2016-05-18 00:57:45,847 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533065566_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:45,847 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533065566_0001_1_01_000000_0
2016-05-18 00:57:45,847 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B04769583C1D4B62B8EC726EB6BABD31, mem on close (mb), free: 133, total: 365, max: 672
2016-05-18 00:57:45,847 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E92CD571D4734E478DD368290DA28EA6
2016-05-18 00:57:45,848 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E92CD571D4734E478DD368290DA28EA6: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:45,848 INFO output.MROutput (MROutput.java:close(526)) - F4C9A4836B984E45821FD5621473B27B closed
2016-05-18 00:57:45,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:45 Completed running task attempt: attempt_1463533065566_0001_1_01_000000_0
2016-05-18 00:57:45,849 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533065566_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:45,849 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533065566_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:45,849 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533065566_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533065839, Process_Duration=8, Process_End_Time=1463533065847, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:45,850 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533065566_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:45,850 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B04769583C1D4B62B8EC726EB6BABD31, taskAttemptId=attempt_1463533065566_0001_1_01_000000_0, creationTime=1463533065803, allocationTime=1463533065804, startTime=1463533065806, finishTime=1463533065850, timeTaken=44, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:45,850 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:45,851 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:45,851 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,851 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533065566_0001_1_01_000000_0
2016-05-18 00:57:45,851 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533065566_0001_00_000002
2016-05-18 00:57:45,851 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:TASK_FINISHED]: vertexName=B04769583C1D4B62B8EC726EB6BABD31, taskId=task_1463533065566_0001_1_01_000000, startTime=1463533065806, finishTime=1463533065851, timeTaken=45, status=SUCCEEDED, successfulAttemptID=attempt_1463533065566_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533065839, Process_Duration=8, Process_End_Time=1463533065847, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:45,852 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:45,852 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31]
2016-05-18 00:57:45,852 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:VERTEX_FINISHED]: vertexName=B04769583C1D4B62B8EC726EB6BABD31, vertexId=vertex_1463533065566_0001_1_01, initRequestedTime=1463533065727, initedTime=1463533065731, startRequestedTime=1463533065740, startedTime=1463533065740, finishTime=1463533065852, timeTaken=112, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533065839, Process_Duration=8, Process_End_Time=1463533065847, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533065806, firstTasksToStart=[ task_1463533065566_0001_1_01_000000 ], lastTaskFinishTime=1463533065850, lastTasksToFinish=[ task_1463533065566_0001_1_01_000000 ], minTaskDuration=44, maxTaskDuration=44, avgTaskDuration=44.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533065566_0001_1_01_000000 ], longestDurationTasks=[ task_1463533065566_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:45,852 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:45,853 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533065566_0001_00_000002, stoppedTime=1463533065853, exitStatus=0
2016-05-18 00:57:45,853 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:45,853 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:45,853 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533065566_0001_1_00 [E92CD571D4734E478DD368290DA28EA6]
2016-05-18 00:57:45,853 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533065566_0001_1
2016-05-18 00:57:45,853 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533065566_0001_1
2016-05-18 00:57:45,853 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533065566_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:45,853 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: F4C9A4836B984E45821FD5621473B27B for vertex: vertex_1463533065566_0001_1_01 [B04769583C1D4B62B8EC726EB6BABD31], outputName: F4C9A4836B984E45821FD5621473B27B
2016-05-18 00:57:45,855 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533065566_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:45,858 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:F4C9A4836B984E45821FD5621473B27B of vertex/vertexGroup:B04769583C1D4B62B8EC726EB6BABD31 isVertexGroupOutput:false
2016-05-18 00:57:45,858 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:45,860 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533065566_0001_1, queueSize=0
2016-05-18 00:57:45,860 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533065566_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533065566_0001_1, startTime=1463533065726, finishTime=1463533065858, timeTaken=132, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=260, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066131627, Process_Duration=10, Process_End_Time=2927066131637, Read_Duration=2, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:45,860 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533065566_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:45,860 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533065566_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:45 Completed Dag: dag_1463533065566_0001_1
2016-05-18 00:57:45,861 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533065566_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:45,861 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533065566_0001_1_post
2016-05-18 00:57:45,861 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:45,861 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:45,861 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:45,861 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5] every/(1/1) ...emblyplatform/0_ARGS_SWAP, with id=dag_1463533065566_0001_1
2016-05-18 00:57:45,861 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5] every/(1/1) ...emblyplatform/0_ARGS_SWAP, with id=dag_1463533065566_0001_1
2016-05-18 00:57:45,862 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533065566_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:45,862 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533065566_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:45,862 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533065566_0001_00_000002, Exiting
2016-05-18 00:57:45,862 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533065566_0001_00_000002
2016-05-18 00:57:45,862 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533065566_0001_00_000002
2016-05-18 00:57:45,862 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533065566_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:45,862 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533065566_0001_00_000002 completed successfully
2016-05-18 00:57:45,867 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...emblyplatform/0_ARGS_SWAP, applicationId=application_1463533065566_0001
2016-05-18 00:57:45,868 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:45,868 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:45,868 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:45,868 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:45,868 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:45,868 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:45,869 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:45,869 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:45,869 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:45,869 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:45,869 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:45,870 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:45,870 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:45,871 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:45,874 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/F029ED616BA841C6A88A3BB2E1EE8B06/AEFBD858C9EE45B4944DFA72A545EDE5/.tez/application_1463533065566_0001
2016-05-18 00:57:45,874 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/0_ARGS_SWAP/_temporary
2016-05-18 00:57:45,874 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.327
2016-05-18 00:57:45,944 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//0_ARGS_SWAP
2016-05-18 00:57:45,969 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:45,969 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:45,998 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:46,006 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 00:57:46,006 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:46,006 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:46,007 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:46,021 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:46,026 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_-1
2016-05-18 00:57:46,083 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:46,083 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:46,084 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_-1"]
2016-05-18 00:57:46,084 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:46,084 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:46,084 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:46,085 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/ALL_'field'_-1
2016-05-18 00:57:46,085 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:46,097 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:46,101 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:46,101 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:46,101 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:46,110 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001 doesn't exist and is created
2016-05-18 00:57:46,123 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:46,123 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:46,124 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001_wd
2016-05-18 00:57:46,133 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533066101_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:46,133 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:46,133 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:46,135 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:46,135 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:46,135 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:46,135 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:46,135 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:46,137 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:46,137 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:46,137 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:46,137 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:46,137 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:46,137 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:46,137 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:46,138 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:46,138 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:46,138 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:46,138 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:46,138 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:46,139 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:46,140 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:46,140 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:46,140 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001_wd/localmode-log-dir
2016-05-18 00:57:46,140 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001_wd/localmode-log-dir/history.txt.appattempt_1463533066101_0001_000000, maxErrors=10
2016-05-18 00:57:46,141 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:46,141 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533066101_0001_000000, appSubmitTime=1463533066133, launchTime=1463533066133
2016-05-18 00:57:46,144 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:46,147 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:51405
2016-05-18 00:57:46,148 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533066101
2016-05-18 00:57:46,150 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533066101_0001_000000, startTime=1463533066149
2016-05-18 00:57:46,150 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:46,223 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:46,224 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:46,224 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/ALL_'field'_-1, applicationId=application_1463533066101_0001, dagName=[C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211] every/(1/1) ...lyplatform/ALL_'field'_-1
2016-05-18 00:57:46,239 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211] every/(1/1) ...lyplatform/ALL_'field'_-1
2016-05-18 00:57:46,239 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:46,240 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533066101_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001_wd/localmode-log-dir/dag_1463533066101_0001_1.dot
2016-05-18 00:57:46,241 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001_wd/localmode-log-dir/dag_1463533066101_0001_1-tez-dag.pb.txt
2016-05-18 00:57:46,243 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:46,243 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:46,247 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533066101_0001_1
2016-05-18 00:57:46,247 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,247 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211] every/(1/1) ...lyplatform/ALL_'field'_-1
2016-05-18 00:57:46 Running Dag: dag_1463533066101_0001_1
2016-05-18 00:57:46,256 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533066101_0001_1, submitTime=1463533066239
2016-05-18 00:57:46,257 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001/tez-conf.pb" } size: 36913 timestamp: 1463533066000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:46,257 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,257 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:46,258 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001/tez-conf.pb" } size: 36913 timestamp: 1463533066000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:46,258 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]
2016-05-18 00:57:46,258 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:46,259 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:46,259 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533066101_0001_1, initTime=1463533066259
2016-05-18 00:57:46,259 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533066101_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:46,260 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:46,260 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:DAG_STARTED]: dagID=dag_1463533066101_0001_1, startTime=1463533066260
2016-05-18 00:57:46,260 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533066101_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:46,260 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: DB3171B8032E4CCFABAD78525DF9F9C5 : {E7A411A87A314A8DACD2E6A63BBAFF78={InputName=E7A411A87A314A8DACD2E6A63BBAFF78}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:46,260 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: E7A411A87A314A8DACD2E6A63BBAFF78, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:46,260 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,261 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,261 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,261 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,261 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:46,261 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]
2016-05-18 00:57:46,261 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: E7A411A87A314A8DACD2E6A63BBAFF78 on vertex vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,262 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/ALL_'field'_-1, applicationId=application_1463533066101_0001, dagName=[C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211] every/(1/1) ...lyplatform/ALL_'field'_-1
2016-05-18 00:57:46,264 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:46,264 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]
2016-05-18 00:57:46,264 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]
2016-05-18 00:57:46,265 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533066101_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:46,265 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40], numAdditionalOutputs=1
2016-05-18 00:57:46,265 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=556A9202E64746EF9F0862DBECF5FD41, vertex=vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:46,266 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input E7A411A87A314A8DACD2E6A63BBAFF78 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:46,267 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E63B6C400A12459BAACE0B9EACFE2E40:556A9202E64746EF9F0862DBECF5FD41 using oldmapred API
2016-05-18 00:57:46,267 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=556A9202E64746EF9F0862DBECF5FD41, vertexName=E63B6C400A12459BAACE0B9EACFE2E40, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:46,268 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:46,268 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:46,268 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:46,269 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E63B6C400A12459BAACE0B9EACFE2E40, vertexId=vertex_1463533066101_0001_1_01, initRequestedTime=1463533066261, initedTime=1463533066265, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:46,269 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:46,269 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:46,269 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: E7A411A87A314A8DACD2E6A63BBAFF78 on vertex vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,270 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:46,270 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {E7A411A87A314A8DACD2E6A63BBAFF78=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:46,271 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] parallelism set to 1
2016-05-18 00:57:46,271 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:46,271 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DB3171B8032E4CCFABAD78525DF9F9C5, vertexId=vertex_1463533066101_0001_1_00, initRequestedTime=1463533066260, initedTime=1463533066271, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:46,271 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] with distanceFromRoot: 0
2016-05-18 00:57:46,271 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:46,271 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533066101_0001_1_00, startRequestedTime=1463533066261, startedTime=1463533066271
2016-05-18 00:57:46,272 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533066101_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:46,272 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:46,272 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533066101_0001_1_00 for vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40] numStartedSources: 1 numSources: 1
2016-05-18 00:57:46,272 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40] with distanceFromRoot: 1
2016-05-18 00:57:46,272 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533066101_0001_1_01, startRequestedTime=1463533066272, startedTime=1463533066272
2016-05-18 00:57:46,272 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:46,275 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E63B6C400A12459BAACE0B9EACFE2E40 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:46,275 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in DB3171B8032E4CCFABAD78525DF9F9C5
2016-05-18 00:57:46,275 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,275 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: DB3171B8032E4CCFABAD78525DF9F9C5 in vertex: E63B6C400A12459BAACE0B9EACFE2E40 numBipartiteSourceTasks: 1
2016-05-18 00:57:46,276 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:TASK_STARTED]: vertexName=DB3171B8032E4CCFABAD78525DF9F9C5, taskId=task_1463533066101_0001_1_00_000000, scheduledTime=1463533066276, launchTime=1463533066276
2016-05-18 00:57:46,276 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,277 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:46,278 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533066101_0001_00_000001, launchTime=1463533066278
2016-05-18 00:57:46,279 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533066101_0001_00_000001
2016-05-18 00:57:46,279 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533066101_0001_1_00_000000_0] started. Is using containerId: [container_1463533066101_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:46,280 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DB3171B8032E4CCFABAD78525DF9F9C5, taskAttemptId=attempt_1463533066101_0001_1_00_000000_0, startTime=1463533066279, containerId=container_1463533066101_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:46,280 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533066101_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533066101_0001_1_00_000000_0
2016-05-18 00:57:46 Starting to run new task attempt: attempt_1463533066101_0001_1_00_000000_0
2016-05-18 00:57:46,281 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533066101_0001_1_00_000000_0
2016-05-18 00:57:46,281 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,282 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:46,283 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211] every/(1/1) ...lyplatform/ALL_'field'_-1, VertexName: DB3171B8032E4CCFABAD78525DF9F9C5, VertexParallelism: 1, TaskAttemptID:attempt_1463533066101_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E7A411A87A314A8DACD2E6A63BBAFF78, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E63B6C400A12459BAACE0B9EACFE2E40, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:46,283 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533066101_0001, with dag id: dag_1463533066101_0001_1
2016-05-18 00:57:46,283 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:46,284 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533066101_0001_1_00_000000_0
2016-05-18 00:57:46,284 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533066101_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:46,284 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533066101_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:46,287 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:46,289 INFO input.MRInput (MRInput.java:initialize(440)) - E7A411A87A314A8DACD2E6A63BBAFF78 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:46,289 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: E7A411A87A314A8DACD2E6A63BBAFF78
2016-05-18 00:57:46,291 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DB3171B8032E4CCFABAD78525DF9F9C5, ordinal: 0
2016-05-18 00:57:46,291 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DB3171B8032E4CCFABAD78525DF9F9C5, mem on start (mb), free: 233, total: 365, max: 672
2016-05-18 00:57:46,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:46,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:46,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:46,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:46,291 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[E63B6C400A12459BAACE0B9EACFE2E40:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [E7A411A87A314A8DACD2E6A63BBAFF78:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 00:57:46,291 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:46,292 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:46,292 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:46,292 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[E63B6C400A12459BAACE0B9EACFE2E40:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [E7A411A87A314A8DACD2E6A63BBAFF78:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 00:57:46,292 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:46,292 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E7A411A87A314A8DACD2E6A63BBAFF78 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:46,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:46,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:46,292 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E7A411A87A314A8DACD2E6A63BBAFF78
2016-05-18 00:57:46,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:46,294 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533066101_0001_1_00_000000_0
2016-05-18 00:57:46,295 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:46,295 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - E7A411A87A314A8DACD2E6A63BBAFF78 initialized RecordReader from event
2016-05-18 00:57:46,297 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:46,297 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: E7A411A87A314A8DACD2E6A63BBAFF78
2016-05-18 00:57:46,297 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 6EF197C4A9624CFF807CE4F2F0B6D349
2016-05-18 00:57:46,297 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 6EF197C4A9624CFF807CE4F2F0B6D349
2016-05-18 00:57:46,298 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:46,301 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:46,301 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E63B6C400A12459BAACE0B9EACFE2E40 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@71f24c5e, comparator=cascading.tuple.hadoop.util.TupleComparator@5a73e9c3, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:46,301 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E63B6C400A12459BAACE0B9EACFE2E40: , UsingHashComparator=false
2016-05-18 00:57:46,314 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:46,314 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:46,314 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E63B6C400A12459BAACE0B9EACFE2E40: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:46,314 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E63B6C400A12459BAACE0B9EACFE2E40: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:46,315 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:46,315 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DB3171B8032E4CCFABAD78525DF9F9C5, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:46,317 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533066101_0001_1_00_000000_0
2016-05-18 00:57:46,317 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DB3171B8032E4CCFABAD78525DF9F9C5, mem on close (mb), free: 131, total: 365, max: 672
2016-05-18 00:57:46,317 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E63B6C400A12459BAACE0B9EACFE2E40: Starting flush of map output
2016-05-18 00:57:46,317 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E63B6C400A12459BAACE0B9EACFE2E40: Span0.length = 20, perItem = 6
2016-05-18 00:57:46,317 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E63B6C400A12459BAACE0B9EACFE2E40: done sorting span=0, length=20, time=0
2016-05-18 00:57:46,318 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E63B6C400A12459BAACE0B9EACFE2E40: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:46,319 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E63B6C400A12459BAACE0B9EACFE2E40: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001_wd/localmode-local-dir/output/attempt_1463533066101_0001_1_00_000000_0_10230_0/file.out
2016-05-18 00:57:46,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:46 Completed running task attempt: attempt_1463533066101_0001_1_00_000000_0
2016-05-18 00:57:46,323 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533066101_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:46,323 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533066101_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:46,323 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533066101_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533066315, Process_Duration=2, Process_End_Time=1463533066317, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:46,324 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533066101_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:46,324 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:46,324 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DB3171B8032E4CCFABAD78525DF9F9C5, taskAttemptId=attempt_1463533066101_0001_1_00_000000_0, creationTime=1463533066276, allocationTime=1463533066278, startTime=1463533066279, finishTime=1463533066324, timeTaken=45, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:46,324 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:46,324 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,324 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533066101_0001_1_00_000000_0
2016-05-18 00:57:46,324 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533066101_0001_00_000001
2016-05-18 00:57:46,325 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:TASK_FINISHED]: vertexName=DB3171B8032E4CCFABAD78525DF9F9C5, taskId=task_1463533066101_0001_1_00_000000, startTime=1463533066279, finishTime=1463533066324, timeTaken=45, status=SUCCEEDED, successfulAttemptID=attempt_1463533066101_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533066315, Process_Duration=2, Process_End_Time=1463533066317, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:46,325 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:46,325 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,326 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:VERTEX_FINISHED]: vertexName=DB3171B8032E4CCFABAD78525DF9F9C5, vertexId=vertex_1463533066101_0001_1_00, initRequestedTime=1463533066260, initedTime=1463533066271, startRequestedTime=1463533066261, startedTime=1463533066271, finishTime=1463533066325, timeTaken=54, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533066315, Process_Duration=2, Process_End_Time=1463533066317, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533066279, firstTasksToStart=[ task_1463533066101_0001_1_00_000000 ], lastTaskFinishTime=1463533066324, lastTasksToFinish=[ task_1463533066101_0001_1_00_000000 ], minTaskDuration=45, maxTaskDuration=45, avgTaskDuration=45.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533066101_0001_1_00_000000 ], longestDurationTasks=[ task_1463533066101_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:46,326 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:46,326 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40] attempt: attempt_1463533066101_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:46,326 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E63B6C400A12459BAACE0B9EACFE2E40
2016-05-18 00:57:46,326 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533066101_0001_00_000001, stoppedTime=1463533066326, exitStatus=0
2016-05-18 00:57:46,326 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=DB3171B8032E4CCFABAD78525DF9F9C5, destinationVertexName=E63B6C400A12459BAACE0B9EACFE2E40. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:46,327 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]
2016-05-18 00:57:46,327 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:46,327 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:46,327 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533066101_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:46,327 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:TASK_STARTED]: vertexName=E63B6C400A12459BAACE0B9EACFE2E40, taskId=task_1463533066101_0001_1_01_000000, scheduledTime=1463533066327, launchTime=1463533066327
2016-05-18 00:57:46,328 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]
2016-05-18 00:57:46,329 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533066101_0001_00_000002, launchTime=1463533066329
2016-05-18 00:57:46,330 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533066101_0001_00_000002
2016-05-18 00:57:46,330 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533066101_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533066101_0001_1_01_000000_0
2016-05-18 00:57:46,330 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533066101_0001_1_01_000000_0] started. Is using containerId: [container_1463533066101_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:46 Starting to run new task attempt: attempt_1463533066101_0001_1_01_000000_0
2016-05-18 00:57:46,330 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E63B6C400A12459BAACE0B9EACFE2E40, taskAttemptId=attempt_1463533066101_0001_1_01_000000_0, startTime=1463533066330, containerId=container_1463533066101_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:46,330 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533066101_0001_1_01_000000_0
2016-05-18 00:57:46,330 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,331 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:46,332 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211] every/(1/1) ...lyplatform/ALL_'field'_-1, VertexName: E63B6C400A12459BAACE0B9EACFE2E40, VertexParallelism: 1, TaskAttemptID:attempt_1463533066101_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DB3171B8032E4CCFABAD78525DF9F9C5, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=556A9202E64746EF9F0862DBECF5FD41, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:46,332 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:46,333 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533066101_0001_1_01_000000_0
2016-05-18 00:57:46,333 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533066101_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]
2016-05-18 00:57:46,335 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533066101_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:46,335 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533066101_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:46,335 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533066101_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:46,335 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533066101_0001_00_000001, Exiting
2016-05-18 00:57:46,335 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533066101_0001_00_000001
2016-05-18 00:57:46,335 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533066101_0001_00_000001
2016-05-18 00:57:46,336 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:46,338 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:46,338 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533066101_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:46,339 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533066101_0001_00_000001 completed successfully
2016-05-18 00:57:46,340 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E63B6C400A12459BAACE0B9EACFE2E40, ordinal: 1
2016-05-18 00:57:46,340 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E63B6C400A12459BAACE0B9EACFE2E40, mem on start (mb), free: 126, total: 365, max: 672
2016-05-18 00:57:46,340 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:46,340 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:46,340 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:46,345 INFO output.MROutput (MROutput.java:initialize(421)) - 556A9202E64746EF9F0862DBECF5FD41: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:46,345 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:46,345 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[556A9202E64746EF9F0862DBECF5FD41:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [DB3171B8032E4CCFABAD78525DF9F9C5:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:46,346 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:46,346 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:46,346 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:46,346 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[556A9202E64746EF9F0862DBECF5FD41:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [DB3171B8032E4CCFABAD78525DF9F9C5:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:46,346 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:46,346 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DB3171B8032E4CCFABAD78525DF9F9C5 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:46,346 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - DB3171B8032E4CCFABAD78525DF9F9C5: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:46,347 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - DB3171B8032E4CCFABAD78525DF9F9C5: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:46,347 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:46,347 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: DB3171B8032E4CCFABAD78525DF9F9C5 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:46,348 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:46,348 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DB3171B8032E4CCFABAD78525DF9F9C5
2016-05-18 00:57:46,349 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:46,349 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533066101_0001_1_01_000000_0
2016-05-18 00:57:46,350 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DB3171B8032E4CCFABAD78525DF9F9C5: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:46,351 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533066101_0001_1_00_000000_0_10230} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533066351, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:46,352 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : DB3171B8032E4CCFABAD78525DF9F9C5
2016-05-18 00:57:46,352 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:46,352 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: DB3171B8032E4CCFABAD78525DF9F9C5, wasInterrupted=false
2016-05-18 00:57:46,353 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:46,353 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:46,353 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:46,354 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 6EF197C4A9624CFF807CE4F2F0B6D349
2016-05-18 00:57:46,354 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_-1"], id: 556A9202E64746EF9F0862DBECF5FD41
2016-05-18 00:57:46,354 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:46,354 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001_wd/localmode-local-dir/output/attempt_1463533066101_0001_1_00_000000_0_10230/file.out, len=181, isLocal=true
2016-05-18 00:57:46,354 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:46,354 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:46,355 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:46,354 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_-1"]
2016-05-18 00:57:46,355 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:46,355 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : DB3171B8032E4CCFABAD78525DF9F9C5
2016-05-18 00:57:46,355 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - DB3171B8032E4CCFABAD78525DF9F9C5: Shuffle Runner thread complete
2016-05-18 00:57:46,355 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 6EF197C4A9624CFF807CE4F2F0B6D349, for 1 inputs
2016-05-18 00:57:46,355 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E63B6C400A12459BAACE0B9EACFE2E40, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:46,356 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:46,357 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - DB3171B8032E4CCFABAD78525DF9F9C5: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:46,361 INFO output.MROutput (MROutput.java:close(526)) - 556A9202E64746EF9F0862DBECF5FD41 closed
2016-05-18 00:57:46,361 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533066101_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:46,362 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533066101_0001_1_01_000000_0
2016-05-18 00:57:46,362 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E63B6C400A12459BAACE0B9EACFE2E40, mem on close (mb), free: 122, total: 365, max: 672
2016-05-18 00:57:46,362 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: DB3171B8032E4CCFABAD78525DF9F9C5
2016-05-18 00:57:46,362 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DB3171B8032E4CCFABAD78525DF9F9C5: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:46,363 INFO output.MROutput (MROutput.java:close(526)) - 556A9202E64746EF9F0862DBECF5FD41 closed
2016-05-18 00:57:46,363 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:46 Completed running task attempt: attempt_1463533066101_0001_1_01_000000_0
2016-05-18 00:57:46,363 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533066101_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:46,364 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533066101_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:46,364 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533066101_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533066356, Process_Duration=6, Process_End_Time=1463533066362, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:46,365 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533066101_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:46,365 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:46,365 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E63B6C400A12459BAACE0B9EACFE2E40, taskAttemptId=attempt_1463533066101_0001_1_01_000000_0, creationTime=1463533066327, allocationTime=1463533066328, startTime=1463533066330, finishTime=1463533066365, timeTaken=35, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:46,365 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533066101_0001_1_01_000000_0
2016-05-18 00:57:46,365 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:TASK_FINISHED]: vertexName=E63B6C400A12459BAACE0B9EACFE2E40, taskId=task_1463533066101_0001_1_01_000000, startTime=1463533066330, finishTime=1463533066365, timeTaken=35, status=SUCCEEDED, successfulAttemptID=attempt_1463533066101_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533066356, Process_Duration=6, Process_End_Time=1463533066362, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:46,366 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:46,366 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40]
2016-05-18 00:57:46,366 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:VERTEX_FINISHED]: vertexName=E63B6C400A12459BAACE0B9EACFE2E40, vertexId=vertex_1463533066101_0001_1_01, initRequestedTime=1463533066261, initedTime=1463533066265, startRequestedTime=1463533066272, startedTime=1463533066272, finishTime=1463533066366, timeTaken=94, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533066356, Process_Duration=6, Process_End_Time=1463533066362, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533066330, firstTasksToStart=[ task_1463533066101_0001_1_01_000000 ], lastTaskFinishTime=1463533066365, lastTasksToFinish=[ task_1463533066101_0001_1_01_000000 ], minTaskDuration=35, maxTaskDuration=35, avgTaskDuration=35.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533066101_0001_1_01_000000 ], longestDurationTasks=[ task_1463533066101_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:46,367 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:46,367 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533066101_0001_00_000002, stoppedTime=1463533066367, exitStatus=0
2016-05-18 00:57:46,367 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:46,367 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:46,367 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533066101_0001_1_00 [DB3171B8032E4CCFABAD78525DF9F9C5]
2016-05-18 00:57:46,367 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533066101_0001_1
2016-05-18 00:57:46,367 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533066101_0001_1
2016-05-18 00:57:46,367 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533066101_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:46,368 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533066101_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:46,365 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:46,370 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,367 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 556A9202E64746EF9F0862DBECF5FD41 for vertex: vertex_1463533066101_0001_1_01 [E63B6C400A12459BAACE0B9EACFE2E40], outputName: 556A9202E64746EF9F0862DBECF5FD41
2016-05-18 00:57:46,370 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533066101_0001_00_000002
2016-05-18 00:57:46,371 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533066101_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:46,371 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533066101_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:46,371 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533066101_0001_00_000002, Exiting
2016-05-18 00:57:46,371 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533066101_0001_00_000002
2016-05-18 00:57:46,371 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533066101_0001_00_000002
2016-05-18 00:57:46,371 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533066101_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:46,371 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533066101_0001_00_000002 completed successfully
2016-05-18 00:57:46,376 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:556A9202E64746EF9F0862DBECF5FD41 of vertex/vertexGroup:E63B6C400A12459BAACE0B9EACFE2E40 isVertexGroupOutput:false
2016-05-18 00:57:46,376 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:46,378 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533066101_0001_1, queueSize=0
2016-05-18 00:57:46,378 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066101_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533066101_0001_1, startTime=1463533066260, finishTime=1463533066376, timeTaken=116, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=190, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066132671, Process_Duration=8, Process_End_Time=2927066132679, Read_Duration=1, Tuples_Read=40, Tuples_Written=40, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:46,378 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533066101_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:46,378 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533066101_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:46 Completed Dag: dag_1463533066101_0001_1
2016-05-18 00:57:46,378 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533066101_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:46,378 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533066101_0001_1_post
2016-05-18 00:57:46,378 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,379 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:46,379 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:46,379 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211] every/(1/1) ...lyplatform/ALL_'field'_-1, with id=dag_1463533066101_0001_1
2016-05-18 00:57:46,379 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211] every/(1/1) ...lyplatform/ALL_'field'_-1, with id=dag_1463533066101_0001_1
2016-05-18 00:57:46,387 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/ALL_'field'_-1, applicationId=application_1463533066101_0001
2016-05-18 00:57:46,388 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:46,389 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:46,389 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:46,389 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:46,389 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:46,389 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:46,389 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:46,389 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:46,389 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:46,389 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:46,390 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:46,390 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:46,390 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:46,391 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:46,393 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C23D1094AFCF4B2D8A510753E0DB636F/C2622CAB44474E37AD07DE6CB519D211/.tez/application_1463533066101_0001
2016-05-18 00:57:46,393 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_-1/_temporary
2016-05-18 00:57:46,394 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.310
2016-05-18 00:57:46,460 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_'field'_-1
2016-05-18 00:57:46,482 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:46,482 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:46,515 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.010
2016-05-18 00:57:46,518 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.013
2016-05-18 00:57:46,518 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:46,518 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:46,518 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:46,543 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:46,548 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_'field'
2016-05-18 00:57:46,592 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:46,592 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:46,592 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_'field'"]
2016-05-18 00:57:46,592 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:46,592 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:46,592 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:46,593 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...tform/ALL_'field'_'field'
2016-05-18 00:57:46,593 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:46,606 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:46,611 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:46,611 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:46,611 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:46,621 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001 doesn't exist and is created
2016-05-18 00:57:46,640 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:46,640 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:46,641 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001_wd
2016-05-18 00:57:46,652 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533066611_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:46,653 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:46,653 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:46,655 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:46,655 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:46,656 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:46,656 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:46,656 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:46,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:46,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:46,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:46,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:46,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:46,658 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:46,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:46,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:46,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:46,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:46,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:46,659 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:46,661 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:46,661 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:46,662 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:46,662 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001_wd/localmode-log-dir
2016-05-18 00:57:46,662 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001_wd/localmode-log-dir/history.txt.appattempt_1463533066611_0001_000000, maxErrors=10
2016-05-18 00:57:46,662 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:46,663 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533066611_0001_000000, appSubmitTime=1463533066651, launchTime=1463533066652
2016-05-18 00:57:46,665 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:46,668 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:52294
2016-05-18 00:57:46,668 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533066611
2016-05-18 00:57:46,673 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533066611_0001_000000, startTime=1463533066673
2016-05-18 00:57:46,673 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:46,740 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:46,740 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:46,740 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...tform/ALL_'field'_'field', applicationId=application_1463533066611_0001, dagName=[1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E] every/(1/1) ...tform/ALL_'field'_'field'
2016-05-18 00:57:46,756 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E] every/(1/1) ...tform/ALL_'field'_'field'
2016-05-18 00:57:46,756 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:46,756 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533066611_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001_wd/localmode-log-dir/dag_1463533066611_0001_1.dot
2016-05-18 00:57:46,757 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001_wd/localmode-log-dir/dag_1463533066611_0001_1-tez-dag.pb.txt
2016-05-18 00:57:46,764 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533066611_0001_1
2016-05-18 00:57:46,764 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,764 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E] every/(1/1) ...tform/ALL_'field'_'field'
2016-05-18 00:57:46 Running Dag: dag_1463533066611_0001_1
2016-05-18 00:57:46,774 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533066611_0001_1, submitTime=1463533066756
2016-05-18 00:57:46,775 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001/tez-conf.pb" } size: 36913 timestamp: 1463533066000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:46,775 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,775 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:46,775 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001/tez-conf.pb" } size: 36913 timestamp: 1463533066000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:46,775 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]
2016-05-18 00:57:46,775 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:46,776 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:46,776 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533066611_0001_1, initTime=1463533066776
2016-05-18 00:57:46,776 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533066611_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:46,776 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:46,776 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:DAG_STARTED]: dagID=dag_1463533066611_0001_1, startTime=1463533066776
2016-05-18 00:57:46,776 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533066611_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:46,776 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: E0B178A16A34444D8854AB6B27C56B71 : {70AD48CAC2E1465DAE36EE19D51F6921={InputName=70AD48CAC2E1465DAE36EE19D51F6921}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:46,776 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 70AD48CAC2E1465DAE36EE19D51F6921, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:46,776 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,777 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,777 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,777 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,777 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:46,777 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 70AD48CAC2E1465DAE36EE19D51F6921 on vertex vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,777 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]
2016-05-18 00:57:46,779 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:46,779 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:46,779 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 70AD48CAC2E1465DAE36EE19D51F6921 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:46,780 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...tform/ALL_'field'_'field', applicationId=application_1463533066611_0001, dagName=[1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E] every/(1/1) ...tform/ALL_'field'_'field'
2016-05-18 00:57:46,781 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:46,781 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]
2016-05-18 00:57:46,782 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]
2016-05-18 00:57:46,782 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533066611_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:46,782 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552], numAdditionalOutputs=1
2016-05-18 00:57:46,782 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=6991BC31A14845CB9D2ADEA76400ADC0, vertex=vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:46,783 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:46,783 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:46,783 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:46,784 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:46,784 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 70AD48CAC2E1465DAE36EE19D51F6921 on vertex vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,785 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D7E1D61F887840E7B5ABEC02998FA552:6991BC31A14845CB9D2ADEA76400ADC0 using oldmapred API
2016-05-18 00:57:46,785 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=6991BC31A14845CB9D2ADEA76400ADC0, vertexName=D7E1D61F887840E7B5ABEC02998FA552, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:46,786 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D7E1D61F887840E7B5ABEC02998FA552, vertexId=vertex_1463533066611_0001_1_01, initRequestedTime=1463533066777, initedTime=1463533066782, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:46,786 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:46,787 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:46,787 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {70AD48CAC2E1465DAE36EE19D51F6921=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:46,787 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] parallelism set to 1
2016-05-18 00:57:46,787 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:46,788 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E0B178A16A34444D8854AB6B27C56B71, vertexId=vertex_1463533066611_0001_1_00, initRequestedTime=1463533066776, initedTime=1463533066788, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:46,789 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] with distanceFromRoot: 0
2016-05-18 00:57:46,790 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:46,791 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533066611_0001_1_00, startRequestedTime=1463533066777, startedTime=1463533066790
2016-05-18 00:57:46,791 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533066611_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:46,791 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:46,791 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533066611_0001_1_00 for vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552] numStartedSources: 1 numSources: 1
2016-05-18 00:57:46,791 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552] with distanceFromRoot: 1
2016-05-18 00:57:46,791 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in E0B178A16A34444D8854AB6B27C56B71
2016-05-18 00:57:46,791 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,792 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533066611_0001_1_01, startRequestedTime=1463533066791, startedTime=1463533066791
2016-05-18 00:57:46,792 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:46,792 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D7E1D61F887840E7B5ABEC02998FA552 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:46,794 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: E0B178A16A34444D8854AB6B27C56B71 in vertex: D7E1D61F887840E7B5ABEC02998FA552 numBipartiteSourceTasks: 1
2016-05-18 00:57:46,794 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:TASK_STARTED]: vertexName=E0B178A16A34444D8854AB6B27C56B71, taskId=task_1463533066611_0001_1_00_000000, scheduledTime=1463533066794, launchTime=1463533066794
2016-05-18 00:57:46,795 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,795 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:46,797 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533066611_0001_00_000001, launchTime=1463533066797
2016-05-18 00:57:46,798 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533066611_0001_00_000001
2016-05-18 00:57:46,798 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533066611_0001_1_00_000000_0] started. Is using containerId: [container_1463533066611_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:46,798 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E0B178A16A34444D8854AB6B27C56B71, taskAttemptId=attempt_1463533066611_0001_1_00_000000_0, startTime=1463533066798, containerId=container_1463533066611_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:46,799 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533066611_0001_00_000001: 1 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533066611_0001_1_00_000000_0
2016-05-18 00:57:46,799 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533066611_0001, with dag id: dag_1463533066611_0001_1
2016-05-18 00:57:46 Starting to run new task attempt: attempt_1463533066611_0001_1_00_000000_0
2016-05-18 00:57:46,799 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533066611_0001_1_00_000000_0
2016-05-18 00:57:46,799 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,799 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:46,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E] every/(1/1) ...tform/ALL_'field'_'field', VertexName: E0B178A16A34444D8854AB6B27C56B71, VertexParallelism: 1, TaskAttemptID:attempt_1463533066611_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=70AD48CAC2E1465DAE36EE19D51F6921, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D7E1D61F887840E7B5ABEC02998FA552, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:46,800 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:46,800 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533066611_0001_1_00_000000_0
2016-05-18 00:57:46,801 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533066611_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:46,802 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533066611_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:46,805 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:46,805 INFO input.MRInput (MRInput.java:initialize(440)) - 70AD48CAC2E1465DAE36EE19D51F6921 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:46,805 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 70AD48CAC2E1465DAE36EE19D51F6921
2016-05-18 00:57:46,808 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E0B178A16A34444D8854AB6B27C56B71, ordinal: 0
2016-05-18 00:57:46,808 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E0B178A16A34444D8854AB6B27C56B71, mem on start (mb), free: 204, total: 351, max: 672
2016-05-18 00:57:46,808 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:46,808 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:46,809 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:46,809 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:46,809 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[70AD48CAC2E1465DAE36EE19D51F6921:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D7E1D61F887840E7B5ABEC02998FA552:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:46,809 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:46,809 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:46,810 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:46,810 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[70AD48CAC2E1465DAE36EE19D51F6921:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D7E1D61F887840E7B5ABEC02998FA552:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:46,810 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:46,810 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 70AD48CAC2E1465DAE36EE19D51F6921 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:46,810 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:46,810 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 70AD48CAC2E1465DAE36EE19D51F6921
2016-05-18 00:57:46,810 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:46,811 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:46,811 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533066611_0001_1_00_000000_0
2016-05-18 00:57:46,812 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:46,812 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 70AD48CAC2E1465DAE36EE19D51F6921 initialized RecordReader from event
2016-05-18 00:57:46,814 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:46,814 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 70AD48CAC2E1465DAE36EE19D51F6921
2016-05-18 00:57:46,814 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: EA9C7AE5138B4199AB65CCEB339B6271
2016-05-18 00:57:46,815 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] EA9C7AE5138B4199AB65CCEB339B6271
2016-05-18 00:57:46,816 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:46,817 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:46,818 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D7E1D61F887840E7B5ABEC02998FA552 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@39d162fa, comparator=cascading.tuple.hadoop.util.TupleComparator@61b928bd, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:46,818 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D7E1D61F887840E7B5ABEC02998FA552: , UsingHashComparator=false
2016-05-18 00:57:46,830 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:46,830 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:46,831 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D7E1D61F887840E7B5ABEC02998FA552: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:46,831 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D7E1D61F887840E7B5ABEC02998FA552: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:46,831 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:46,831 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E0B178A16A34444D8854AB6B27C56B71, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:46,835 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533066611_0001_1_00_000000_0
2016-05-18 00:57:46,835 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E0B178A16A34444D8854AB6B27C56B71, mem on close (mb), free: 102, total: 351, max: 672
2016-05-18 00:57:46,835 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D7E1D61F887840E7B5ABEC02998FA552: Starting flush of map output
2016-05-18 00:57:46,836 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D7E1D61F887840E7B5ABEC02998FA552: Span0.length = 20, perItem = 6
2016-05-18 00:57:46,836 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D7E1D61F887840E7B5ABEC02998FA552: done sorting span=0, length=20, time=0
2016-05-18 00:57:46,836 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D7E1D61F887840E7B5ABEC02998FA552: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:46,838 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D7E1D61F887840E7B5ABEC02998FA552: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001_wd/localmode-local-dir/output/attempt_1463533066611_0001_1_00_000000_0_10237_0/file.out
2016-05-18 00:57:46,841 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:46 Completed running task attempt: attempt_1463533066611_0001_1_00_000000_0
2016-05-18 00:57:46,842 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533066611_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:46,843 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533066611_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:46,843 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533066611_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533066832, Process_Duration=3, Process_End_Time=1463533066835, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:46,844 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533066611_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:46,844 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:46,844 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E0B178A16A34444D8854AB6B27C56B71, taskAttemptId=attempt_1463533066611_0001_1_00_000000_0, creationTime=1463533066794, allocationTime=1463533066796, startTime=1463533066798, finishTime=1463533066844, timeTaken=46, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:46,844 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:46,844 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,844 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533066611_0001_1_00_000000_0
2016-05-18 00:57:46,845 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:TASK_FINISHED]: vertexName=E0B178A16A34444D8854AB6B27C56B71, taskId=task_1463533066611_0001_1_00_000000, startTime=1463533066798, finishTime=1463533066844, timeTaken=46, status=SUCCEEDED, successfulAttemptID=attempt_1463533066611_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533066832, Process_Duration=3, Process_End_Time=1463533066835, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:46,845 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533066611_0001_00_000001
2016-05-18 00:57:46,846 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:46,846 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,846 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:VERTEX_FINISHED]: vertexName=E0B178A16A34444D8854AB6B27C56B71, vertexId=vertex_1463533066611_0001_1_00, initRequestedTime=1463533066776, initedTime=1463533066788, startRequestedTime=1463533066777, startedTime=1463533066790, finishTime=1463533066846, timeTaken=56, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533066832, Process_Duration=3, Process_End_Time=1463533066835, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533066798, firstTasksToStart=[ task_1463533066611_0001_1_00_000000 ], lastTaskFinishTime=1463533066844, lastTasksToFinish=[ task_1463533066611_0001_1_00_000000 ], minTaskDuration=46, maxTaskDuration=46, avgTaskDuration=46.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533066611_0001_1_00_000000 ], longestDurationTasks=[ task_1463533066611_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:46,846 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:46,846 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533066611_0001_00_000001, stoppedTime=1463533066846, exitStatus=0
2016-05-18 00:57:46,846 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552] attempt: attempt_1463533066611_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:46,847 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:46,847 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:46,847 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D7E1D61F887840E7B5ABEC02998FA552
2016-05-18 00:57:46,847 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533066611_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:46,847 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=E0B178A16A34444D8854AB6B27C56B71, destinationVertexName=D7E1D61F887840E7B5ABEC02998FA552. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:46,847 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]
2016-05-18 00:57:46,847 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:TASK_STARTED]: vertexName=D7E1D61F887840E7B5ABEC02998FA552, taskId=task_1463533066611_0001_1_01_000000, scheduledTime=1463533066847, launchTime=1463533066847
2016-05-18 00:57:46,848 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]
2016-05-18 00:57:46,849 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533066611_0001_00_000002, launchTime=1463533066849
2016-05-18 00:57:46,849 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533066611_0001_00_000002
2016-05-18 00:57:46,850 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533066611_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533066611_0001_1_01_000000_0
2016-05-18 00:57:46,850 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533066611_0001_1_01_000000_0] started. Is using containerId: [container_1463533066611_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:46,850 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D7E1D61F887840E7B5ABEC02998FA552, taskAttemptId=attempt_1463533066611_0001_1_01_000000_0, startTime=1463533066850, containerId=container_1463533066611_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:46 Starting to run new task attempt: attempt_1463533066611_0001_1_01_000000_0
2016-05-18 00:57:46,850 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533066611_0001_1_01_000000_0
2016-05-18 00:57:46,851 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,851 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:46,851 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E] every/(1/1) ...tform/ALL_'field'_'field', VertexName: D7E1D61F887840E7B5ABEC02998FA552, VertexParallelism: 1, TaskAttemptID:attempt_1463533066611_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E0B178A16A34444D8854AB6B27C56B71, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=6991BC31A14845CB9D2ADEA76400ADC0, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:46,851 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:46,852 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533066611_0001_1_01_000000_0
2016-05-18 00:57:46,853 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533066611_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]
2016-05-18 00:57:46,853 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533066611_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:46,855 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:46,855 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:46,859 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D7E1D61F887840E7B5ABEC02998FA552, ordinal: 1
2016-05-18 00:57:46,859 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D7E1D61F887840E7B5ABEC02998FA552, mem on start (mb), free: 97, total: 351, max: 672
2016-05-18 00:57:46,859 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:46,859 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:46,859 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:46,859 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533066611_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:46,859 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533066611_0001_00_000001: 14 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:46,859 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533066611_0001_00_000001, Exiting
2016-05-18 00:57:46,859 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533066611_0001_00_000001
2016-05-18 00:57:46,859 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533066611_0001_00_000001
2016-05-18 00:57:46,860 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533066611_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:46,860 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533066611_0001_00_000001 completed successfully
2016-05-18 00:57:46,864 INFO output.MROutput (MROutput.java:initialize(421)) - 6991BC31A14845CB9D2ADEA76400ADC0: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:46,864 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:46,864 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[E0B178A16A34444D8854AB6B27C56B71:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [6991BC31A14845CB9D2ADEA76400ADC0:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:46,864 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:46,865 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:46,865 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:46,865 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[E0B178A16A34444D8854AB6B27C56B71:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [6991BC31A14845CB9D2ADEA76400ADC0:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:46,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:46,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: E0B178A16A34444D8854AB6B27C56B71 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:46,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:46,865 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:46,866 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - E0B178A16A34444D8854AB6B27C56B71: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:46,866 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - E0B178A16A34444D8854AB6B27C56B71: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:46,866 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: E0B178A16A34444D8854AB6B27C56B71 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:46,867 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: E0B178A16A34444D8854AB6B27C56B71
2016-05-18 00:57:46,867 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:46,867 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533066611_0001_1_01_000000_0
2016-05-18 00:57:46,870 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E0B178A16A34444D8854AB6B27C56B71: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:46,871 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533066611_0001_1_00_000000_0_10237} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533066871, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:46,871 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : E0B178A16A34444D8854AB6B27C56B71
2016-05-18 00:57:46,872 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:46,872 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:46,872 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: E0B178A16A34444D8854AB6B27C56B71, wasInterrupted=false
2016-05-18 00:57:46,872 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: EA9C7AE5138B4199AB65CCEB339B6271
2016-05-18 00:57:46,872 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:46,873 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_'field'"], id: 6991BC31A14845CB9D2ADEA76400ADC0
2016-05-18 00:57:46,873 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_'field'"]
2016-05-18 00:57:46,873 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] EA9C7AE5138B4199AB65CCEB339B6271, for 1 inputs
2016-05-18 00:57:46,873 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:46,874 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:46,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001_wd/localmode-local-dir/output/attempt_1463533066611_0001_1_00_000000_0_10237/file.out, len=181, isLocal=true
2016-05-18 00:57:46,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:46,874 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:46,874 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:46,875 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:46,875 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : E0B178A16A34444D8854AB6B27C56B71
2016-05-18 00:57:46,875 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - E0B178A16A34444D8854AB6B27C56B71: Shuffle Runner thread complete
2016-05-18 00:57:46,875 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D7E1D61F887840E7B5ABEC02998FA552, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:57:46,876 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:46,876 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - E0B178A16A34444D8854AB6B27C56B71: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:46,881 INFO output.MROutput (MROutput.java:close(526)) - 6991BC31A14845CB9D2ADEA76400ADC0 closed
2016-05-18 00:57:46,882 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533066611_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:46,883 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533066611_0001_1_01_000000_0
2016-05-18 00:57:46,883 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D7E1D61F887840E7B5ABEC02998FA552, mem on close (mb), free: 92, total: 351, max: 672
2016-05-18 00:57:46,883 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: E0B178A16A34444D8854AB6B27C56B71
2016-05-18 00:57:46,883 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - E0B178A16A34444D8854AB6B27C56B71: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:46,883 INFO output.MROutput (MROutput.java:close(526)) - 6991BC31A14845CB9D2ADEA76400ADC0 closed
2016-05-18 00:57:46,883 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:46 Completed running task attempt: attempt_1463533066611_0001_1_01_000000_0
2016-05-18 00:57:46,884 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533066611_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:46,884 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533066611_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:46,884 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533066611_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533066875, Process_Duration=8, Process_End_Time=1463533066883, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:46,885 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533066611_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:46,885 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:46,885 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:46,885 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,885 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D7E1D61F887840E7B5ABEC02998FA552, taskAttemptId=attempt_1463533066611_0001_1_01_000000_0, creationTime=1463533066848, allocationTime=1463533066848, startTime=1463533066850, finishTime=1463533066885, timeTaken=35, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:46,885 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533066611_0001_00_000002
2016-05-18 00:57:46,885 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533066611_0001_1_01_000000_0
2016-05-18 00:57:46,886 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:TASK_FINISHED]: vertexName=D7E1D61F887840E7B5ABEC02998FA552, taskId=task_1463533066611_0001_1_01_000000, startTime=1463533066850, finishTime=1463533066886, timeTaken=36, status=SUCCEEDED, successfulAttemptID=attempt_1463533066611_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533066875, Process_Duration=8, Process_End_Time=1463533066883, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:46,886 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:46,886 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552]
2016-05-18 00:57:46,887 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:VERTEX_FINISHED]: vertexName=D7E1D61F887840E7B5ABEC02998FA552, vertexId=vertex_1463533066611_0001_1_01, initRequestedTime=1463533066777, initedTime=1463533066782, startRequestedTime=1463533066791, startedTime=1463533066791, finishTime=1463533066886, timeTaken=95, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533066875, Process_Duration=8, Process_End_Time=1463533066883, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533066850, firstTasksToStart=[ task_1463533066611_0001_1_01_000000 ], lastTaskFinishTime=1463533066885, lastTasksToFinish=[ task_1463533066611_0001_1_01_000000 ], minTaskDuration=35, maxTaskDuration=35, avgTaskDuration=35.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533066611_0001_1_01_000000 ], longestDurationTasks=[ task_1463533066611_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:46,887 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:46,887 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533066611_0001_00_000002, stoppedTime=1463533066887, exitStatus=0
2016-05-18 00:57:46,887 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:46,887 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:46,887 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533066611_0001_1_00 [E0B178A16A34444D8854AB6B27C56B71]
2016-05-18 00:57:46,888 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533066611_0001_1
2016-05-18 00:57:46,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533066611_0001_1
2016-05-18 00:57:46,888 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533066611_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:46,888 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 6991BC31A14845CB9D2ADEA76400ADC0 for vertex: vertex_1463533066611_0001_1_01 [D7E1D61F887840E7B5ABEC02998FA552], outputName: 6991BC31A14845CB9D2ADEA76400ADC0
2016-05-18 00:57:46,888 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533066611_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:46,894 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:6991BC31A14845CB9D2ADEA76400ADC0 of vertex/vertexGroup:D7E1D61F887840E7B5ABEC02998FA552 isVertexGroupOutput:false
2016-05-18 00:57:46,894 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:46,895 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533066611_0001_1, queueSize=0
2016-05-18 00:57:46,895 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533066611_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533066611_0001_1, startTime=1463533066776, finishTime=1463533066894, timeTaken=118, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=190, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066133707, Process_Duration=11, Process_End_Time=2927066133718, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:46,895 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533066611_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:46,896 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533066611_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:46,896 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533066611_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:46 Completed Dag: dag_1463533066611_0001_1
2016-05-18 00:57:46,896 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533066611_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:46,896 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533066611_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:46,896 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533066611_0001_00_000002, Exiting
2016-05-18 00:57:46,896 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533066611_0001_00_000002
2016-05-18 00:57:46,896 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533066611_0001_00_000002
2016-05-18 00:57:46,896 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533066611_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:46,896 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533066611_0001_00_000002 completed successfully
2016-05-18 00:57:46,896 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533066611_0001_1_post
2016-05-18 00:57:46,897 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:46,897 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:46,897 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:46,897 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E] every/(1/1) ...tform/ALL_'field'_'field', with id=dag_1463533066611_0001_1
2016-05-18 00:57:46,897 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E] every/(1/1) ...tform/ALL_'field'_'field', with id=dag_1463533066611_0001_1
2016-05-18 00:57:46,912 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...tform/ALL_'field'_'field', applicationId=application_1463533066611_0001
2016-05-18 00:57:46,913 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:46,913 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:46,913 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:46,913 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:46,913 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:46,914 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:46,914 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:46,914 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:46,914 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:46,915 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:46,915 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:46,915 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:46,915 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:46,916 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:46,920 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1CBDED2481424A92B452A04767589D81/D5B8F04703194F1EB969353B2DEDEE2E/.tez/application_1463533066611_0001
2016-05-18 00:57:46,921 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_'field'/_temporary
2016-05-18 00:57:46,921 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.330
2016-05-18 00:57:46,985 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_'field'_'field'
2016-05-18 00:57:47,013 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:47,014 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:47,042 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.005
2016-05-18 00:57:47,049 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 00:57:47,050 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:47,050 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:47,050 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:47,068 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:47,075 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_RESULTS
2016-05-18 00:57:47,142 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:47,142 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:47,142 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_RESULTS"]
2016-05-18 00:57:47,143 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:47,143 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:47,144 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:47,145 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...tform/ALL_'field'_RESULTS
2016-05-18 00:57:47,146 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:47,164 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:47,168 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:47,168 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:47,168 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:47,177 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001 doesn't exist and is created
2016-05-18 00:57:47,190 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:47,190 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:47,191 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001_wd
2016-05-18 00:57:47,201 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533067168_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:47,201 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:47,201 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:47,203 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:47,203 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:47,203 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:47,203 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:47,204 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:47,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:47,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:47,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:47,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:47,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:47,206 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:47,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:47,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:47,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:47,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:47,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:47,206 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:47,208 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:47,209 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:47,209 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:47,209 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001_wd/localmode-log-dir
2016-05-18 00:57:47,209 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001_wd/localmode-log-dir/history.txt.appattempt_1463533067168_0001_000000, maxErrors=10
2016-05-18 00:57:47,210 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:47,210 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533067168_0001_000000, appSubmitTime=1463533067200, launchTime=1463533067200
2016-05-18 00:57:47,212 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:47,217 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:60625
2016-05-18 00:57:47,217 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533067168
2016-05-18 00:57:47,218 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533067168_0001_000000, startTime=1463533067218
2016-05-18 00:57:47,218 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:47,291 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:47,291 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:47,291 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...tform/ALL_'field'_RESULTS, applicationId=application_1463533067168_0001, dagName=[6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914] every/(1/1) ...tform/ALL_'field'_RESULTS
2016-05-18 00:57:47,307 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914] every/(1/1) ...tform/ALL_'field'_RESULTS
2016-05-18 00:57:47,307 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:47,308 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533067168_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001_wd/localmode-log-dir/dag_1463533067168_0001_1.dot
2016-05-18 00:57:47,308 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001_wd/localmode-log-dir/dag_1463533067168_0001_1-tez-dag.pb.txt
2016-05-18 00:57:47,314 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533067168_0001_1
2016-05-18 00:57:47,314 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,314 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914] every/(1/1) ...tform/ALL_'field'_RESULTS
2016-05-18 00:57:47 Running Dag: dag_1463533067168_0001_1
2016-05-18 00:57:47,322 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533067168_0001_1, submitTime=1463533067307
2016-05-18 00:57:47,324 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001/tez-conf.pb" } size: 36913 timestamp: 1463533067000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:47,324 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,324 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:47,325 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001/tez-conf.pb" } size: 36913 timestamp: 1463533067000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:47,325 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]
2016-05-18 00:57:47,325 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:47,326 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:47,326 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533067168_0001_1, initTime=1463533067326
2016-05-18 00:57:47,326 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533067168_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:47,327 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:47,327 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:DAG_STARTED]: dagID=dag_1463533067168_0001_1, startTime=1463533067327
2016-05-18 00:57:47,327 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533067168_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:47,327 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: EE29D0DC0636447AB3F136BE2A9F415F : {D9C1BD6A8EF74914B49E147021EEC0B1={InputName=D9C1BD6A8EF74914B49E147021EEC0B1}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:47,327 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: D9C1BD6A8EF74914B49E147021EEC0B1, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:47,327 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,327 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,327 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,327 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,328 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:47,328 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]
2016-05-18 00:57:47,328 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...tform/ALL_'field'_RESULTS, applicationId=application_1463533067168_0001, dagName=[6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914] every/(1/1) ...tform/ALL_'field'_RESULTS
2016-05-18 00:57:47,330 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:47,330 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]
2016-05-18 00:57:47,330 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]
2016-05-18 00:57:47,331 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533067168_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:47,331 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1], numAdditionalOutputs=1
2016-05-18 00:57:47,331 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=34D59C56662C445DA574320834D63DFA, vertex=vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:47,331 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: D9C1BD6A8EF74914B49E147021EEC0B1 on vertex vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,333 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for E077277B6A994948AFF227AB8198B4A1:34D59C56662C445DA574320834D63DFA using oldmapred API
2016-05-18 00:57:47,333 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=34D59C56662C445DA574320834D63DFA, vertexName=E077277B6A994948AFF227AB8198B4A1, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:47,333 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input D9C1BD6A8EF74914B49E147021EEC0B1 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:47,333 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E077277B6A994948AFF227AB8198B4A1, vertexId=vertex_1463533067168_0001_1_01, initRequestedTime=1463533067328, initedTime=1463533067331, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:47,333 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:47,334 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:47,334 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:47,335 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:47,335 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:47,335 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: D9C1BD6A8EF74914B49E147021EEC0B1 on vertex vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,336 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:47,336 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {D9C1BD6A8EF74914B49E147021EEC0B1=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:47,336 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] parallelism set to 1
2016-05-18 00:57:47,336 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:47,337 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EE29D0DC0636447AB3F136BE2A9F415F, vertexId=vertex_1463533067168_0001_1_00, initRequestedTime=1463533067327, initedTime=1463533067337, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:47,337 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] with distanceFromRoot: 0
2016-05-18 00:57:47,337 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:47,337 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533067168_0001_1_00, startRequestedTime=1463533067328, startedTime=1463533067337
2016-05-18 00:57:47,338 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533067168_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:47,338 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:47,338 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533067168_0001_1_00 for vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1] numStartedSources: 1 numSources: 1
2016-05-18 00:57:47,338 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1] with distanceFromRoot: 1
2016-05-18 00:57:47,338 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in EE29D0DC0636447AB3F136BE2A9F415F
2016-05-18 00:57:47,339 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,339 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533067168_0001_1_01, startRequestedTime=1463533067338, startedTime=1463533067339
2016-05-18 00:57:47,339 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:47,340 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: E077277B6A994948AFF227AB8198B4A1 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:47,340 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:TASK_STARTED]: vertexName=EE29D0DC0636447AB3F136BE2A9F415F, taskId=task_1463533067168_0001_1_00_000000, scheduledTime=1463533067340, launchTime=1463533067340
2016-05-18 00:57:47,340 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: EE29D0DC0636447AB3F136BE2A9F415F in vertex: E077277B6A994948AFF227AB8198B4A1 numBipartiteSourceTasks: 1
2016-05-18 00:57:47,341 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,341 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:47,343 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533067168_0001_00_000001, launchTime=1463533067342
2016-05-18 00:57:47,343 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533067168_0001_00_000001
2016-05-18 00:57:47,343 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533067168_0001_1_00_000000_0] started. Is using containerId: [container_1463533067168_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:47,344 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EE29D0DC0636447AB3F136BE2A9F415F, taskAttemptId=attempt_1463533067168_0001_1_00_000000_0, startTime=1463533067343, containerId=container_1463533067168_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:47,344 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533067168_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533067168_0001_1_00_000000_0
2016-05-18 00:57:47 Starting to run new task attempt: attempt_1463533067168_0001_1_00_000000_0
2016-05-18 00:57:47,345 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533067168_0001_1_00_000000_0
2016-05-18 00:57:47,345 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,346 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:47,346 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914] every/(1/1) ...tform/ALL_'field'_RESULTS, VertexName: EE29D0DC0636447AB3F136BE2A9F415F, VertexParallelism: 1, TaskAttemptID:attempt_1463533067168_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D9C1BD6A8EF74914B49E147021EEC0B1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E077277B6A994948AFF227AB8198B4A1, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:47,346 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:47,347 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533067168_0001_1_00_000000_0
2016-05-18 00:57:47,347 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533067168_0001, with dag id: dag_1463533067168_0001_1
2016-05-18 00:57:47,347 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533067168_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:47,348 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533067168_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:47,350 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:47,354 INFO input.MRInput (MRInput.java:initialize(440)) - D9C1BD6A8EF74914B49E147021EEC0B1 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:47,355 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EE29D0DC0636447AB3F136BE2A9F415F, ordinal: 0
2016-05-18 00:57:47,355 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EE29D0DC0636447AB3F136BE2A9F415F, mem on start (mb), free: 203, total: 365, max: 672
2016-05-18 00:57:47,355 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:47,355 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:47,355 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:47,355 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: D9C1BD6A8EF74914B49E147021EEC0B1
2016-05-18 00:57:47,356 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:47,356 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D9C1BD6A8EF74914B49E147021EEC0B1:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [E077277B6A994948AFF227AB8198B4A1:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:47,356 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:47,356 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:47,357 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:47,357 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D9C1BD6A8EF74914B49E147021EEC0B1:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [E077277B6A994948AFF227AB8198B4A1:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:47,357 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:47,357 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D9C1BD6A8EF74914B49E147021EEC0B1 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:47,357 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D9C1BD6A8EF74914B49E147021EEC0B1
2016-05-18 00:57:47,357 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:47,358 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:47,358 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:47,358 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533067168_0001_1_00_000000_0
2016-05-18 00:57:47,359 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:47,359 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:47,360 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:47,360 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - D9C1BD6A8EF74914B49E147021EEC0B1 initialized RecordReader from event
2016-05-18 00:57:47,361 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:47,361 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: D9C1BD6A8EF74914B49E147021EEC0B1
2016-05-18 00:57:47,362 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: F5F930FCCE004C7B83EEFAE95C0F937C
2016-05-18 00:57:47,362 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] F5F930FCCE004C7B83EEFAE95C0F937C
2016-05-18 00:57:47,363 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:47,366 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:47,367 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - E077277B6A994948AFF227AB8198B4A1 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4efaa00b, comparator=cascading.tuple.hadoop.util.TupleComparator@19835a53, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:47,367 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for E077277B6A994948AFF227AB8198B4A1: , UsingHashComparator=false
2016-05-18 00:57:47,381 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:47,381 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:47,381 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for E077277B6A994948AFF227AB8198B4A1: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:47,381 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - E077277B6A994948AFF227AB8198B4A1: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:47,381 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:47,381 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EE29D0DC0636447AB3F136BE2A9F415F, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:47,383 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533067168_0001_1_00_000000_0
2016-05-18 00:57:47,383 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EE29D0DC0636447AB3F136BE2A9F415F, mem on close (mb), free: 100, total: 365, max: 672
2016-05-18 00:57:47,383 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - E077277B6A994948AFF227AB8198B4A1: Starting flush of map output
2016-05-18 00:57:47,383 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - E077277B6A994948AFF227AB8198B4A1: Span0.length = 20, perItem = 6
2016-05-18 00:57:47,383 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - E077277B6A994948AFF227AB8198B4A1: done sorting span=0, length=20, time=0
2016-05-18 00:57:47,384 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - E077277B6A994948AFF227AB8198B4A1: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:47,384 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - E077277B6A994948AFF227AB8198B4A1: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001_wd/localmode-local-dir/output/attempt_1463533067168_0001_1_00_000000_0_10243_0/file.out
2016-05-18 00:57:47,387 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:47 Completed running task attempt: attempt_1463533067168_0001_1_00_000000_0
2016-05-18 00:57:47,388 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533067168_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:47,388 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533067168_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:47,388 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533067168_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533067381, Process_Duration=2, Process_End_Time=1463533067383, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:47,389 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533067168_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:47,389 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:47,389 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:47,389 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,389 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EE29D0DC0636447AB3F136BE2A9F415F, taskAttemptId=attempt_1463533067168_0001_1_00_000000_0, creationTime=1463533067341, allocationTime=1463533067342, startTime=1463533067343, finishTime=1463533067389, timeTaken=46, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:47,389 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533067168_0001_00_000001
2016-05-18 00:57:47,389 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533067168_0001_1_00_000000_0
2016-05-18 00:57:47,390 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:TASK_FINISHED]: vertexName=EE29D0DC0636447AB3F136BE2A9F415F, taskId=task_1463533067168_0001_1_00_000000, startTime=1463533067343, finishTime=1463533067389, timeTaken=46, status=SUCCEEDED, successfulAttemptID=attempt_1463533067168_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533067381, Process_Duration=2, Process_End_Time=1463533067383, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:47,390 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:47,390 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,391 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:VERTEX_FINISHED]: vertexName=EE29D0DC0636447AB3F136BE2A9F415F, vertexId=vertex_1463533067168_0001_1_00, initRequestedTime=1463533067327, initedTime=1463533067337, startRequestedTime=1463533067328, startedTime=1463533067337, finishTime=1463533067390, timeTaken=53, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533067381, Process_Duration=2, Process_End_Time=1463533067383, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533067343, firstTasksToStart=[ task_1463533067168_0001_1_00_000000 ], lastTaskFinishTime=1463533067389, lastTasksToFinish=[ task_1463533067168_0001_1_00_000000 ], minTaskDuration=46, maxTaskDuration=46, avgTaskDuration=46.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533067168_0001_1_00_000000 ], longestDurationTasks=[ task_1463533067168_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:47,391 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:47,391 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533067168_0001_00_000001, stoppedTime=1463533067391, exitStatus=0
2016-05-18 00:57:47,391 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1] attempt: attempt_1463533067168_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:47,391 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:47,392 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:47,391 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E077277B6A994948AFF227AB8198B4A1
2016-05-18 00:57:47,392 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=EE29D0DC0636447AB3F136BE2A9F415F, destinationVertexName=E077277B6A994948AFF227AB8198B4A1. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:47,392 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]
2016-05-18 00:57:47,392 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533067168_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:47,392 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:TASK_STARTED]: vertexName=E077277B6A994948AFF227AB8198B4A1, taskId=task_1463533067168_0001_1_01_000000, scheduledTime=1463533067392, launchTime=1463533067392
2016-05-18 00:57:47,392 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]
2016-05-18 00:57:47,395 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533067168_0001_00_000002, launchTime=1463533067395
2016-05-18 00:57:47,395 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533067168_0001_00_000002
2016-05-18 00:57:47,396 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533067168_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533067168_0001_1_01_000000_0
2016-05-18 00:57:47,396 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533067168_0001_1_01_000000_0] started. Is using containerId: [container_1463533067168_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:47,396 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E077277B6A994948AFF227AB8198B4A1, taskAttemptId=attempt_1463533067168_0001_1_01_000000_0, startTime=1463533067396, containerId=container_1463533067168_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:47 Starting to run new task attempt: attempt_1463533067168_0001_1_01_000000_0
2016-05-18 00:57:47,396 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533067168_0001_1_01_000000_0
2016-05-18 00:57:47,396 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,397 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:47,397 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914] every/(1/1) ...tform/ALL_'field'_RESULTS, VertexName: E077277B6A994948AFF227AB8198B4A1, VertexParallelism: 1, TaskAttemptID:attempt_1463533067168_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EE29D0DC0636447AB3F136BE2A9F415F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=34D59C56662C445DA574320834D63DFA, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:47,397 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:47,398 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533067168_0001_1_01_000000_0
2016-05-18 00:57:47,398 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533067168_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]
2016-05-18 00:57:47,398 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533067168_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:47,400 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:47,400 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533067168_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:47,400 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533067168_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:47,401 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:47,401 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533067168_0001_00_000001, Exiting
2016-05-18 00:57:47,401 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533067168_0001_00_000001
2016-05-18 00:57:47,401 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533067168_0001_00_000001
2016-05-18 00:57:47,402 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533067168_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:47,402 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533067168_0001_00_000001 completed successfully
2016-05-18 00:57:47,406 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E077277B6A994948AFF227AB8198B4A1, ordinal: 1
2016-05-18 00:57:47,406 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E077277B6A994948AFF227AB8198B4A1, mem on start (mb), free: 94, total: 365, max: 672
2016-05-18 00:57:47,406 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:47,406 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:47,406 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:47,410 INFO output.MROutput (MROutput.java:initialize(421)) - 34D59C56662C445DA574320834D63DFA: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:47,410 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:47,410 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[EE29D0DC0636447AB3F136BE2A9F415F:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [34D59C56662C445DA574320834D63DFA:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:47,410 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:47,410 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:47,410 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:47,410 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[EE29D0DC0636447AB3F136BE2A9F415F:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [34D59C56662C445DA574320834D63DFA:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:47,411 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:47,411 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: EE29D0DC0636447AB3F136BE2A9F415F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:47,411 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:47,411 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:47,411 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - EE29D0DC0636447AB3F136BE2A9F415F: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:47,411 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - EE29D0DC0636447AB3F136BE2A9F415F: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:47,412 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: EE29D0DC0636447AB3F136BE2A9F415F with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:47,413 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: EE29D0DC0636447AB3F136BE2A9F415F
2016-05-18 00:57:47,413 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:47,413 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533067168_0001_1_01_000000_0
2016-05-18 00:57:47,413 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EE29D0DC0636447AB3F136BE2A9F415F: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:47,415 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533067168_0001_1_00_000000_0_10243} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533067415, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:47,415 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : EE29D0DC0636447AB3F136BE2A9F415F
2016-05-18 00:57:47,415 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:47,415 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: EE29D0DC0636447AB3F136BE2A9F415F, wasInterrupted=false
2016-05-18 00:57:47,416 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:47,416 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:47,416 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - EE29D0DC0636447AB3F136BE2A9F415F: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:47,416 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:47,416 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: F5F930FCCE004C7B83EEFAE95C0F937C
2016-05-18 00:57:47,417 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_RESULTS"], id: 34D59C56662C445DA574320834D63DFA
2016-05-18 00:57:47,417 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_RESULTS"]
2016-05-18 00:57:47,417 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] F5F930FCCE004C7B83EEFAE95C0F937C, for 1 inputs
2016-05-18 00:57:47,418 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:47,418 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001_wd/localmode-local-dir/output/attempt_1463533067168_0001_1_00_000000_0_10243/file.out, len=181, isLocal=true
2016-05-18 00:57:47,418 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:47,418 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:47,418 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:47,418 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:47,418 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : EE29D0DC0636447AB3F136BE2A9F415F
2016-05-18 00:57:47,418 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - EE29D0DC0636447AB3F136BE2A9F415F: Shuffle Runner thread complete
2016-05-18 00:57:47,418 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E077277B6A994948AFF227AB8198B4A1, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:57:47,419 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:47,420 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - EE29D0DC0636447AB3F136BE2A9F415F: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:47,423 INFO output.MROutput (MROutput.java:close(526)) - 34D59C56662C445DA574320834D63DFA closed
2016-05-18 00:57:47,423 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533067168_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:47,426 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533067168_0001_1_01_000000_0
2016-05-18 00:57:47,426 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E077277B6A994948AFF227AB8198B4A1, mem on close (mb), free: 90, total: 365, max: 672
2016-05-18 00:57:47,426 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: EE29D0DC0636447AB3F136BE2A9F415F
2016-05-18 00:57:47,427 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - EE29D0DC0636447AB3F136BE2A9F415F: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:47,427 INFO output.MROutput (MROutput.java:close(526)) - 34D59C56662C445DA574320834D63DFA closed
2016-05-18 00:57:47,427 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:47 Completed running task attempt: attempt_1463533067168_0001_1_01_000000_0
2016-05-18 00:57:47,427 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533067168_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:47,427 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533067168_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:47,427 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533067168_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533067419, Process_Duration=7, Process_End_Time=1463533067426, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:47,428 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533067168_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:47,428 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:47,428 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:47,428 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,428 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E077277B6A994948AFF227AB8198B4A1, taskAttemptId=attempt_1463533067168_0001_1_01_000000_0, creationTime=1463533067392, allocationTime=1463533067394, startTime=1463533067396, finishTime=1463533067428, timeTaken=32, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:47,429 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533067168_0001_1_01_000000_0
2016-05-18 00:57:47,429 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533067168_0001_00_000002
2016-05-18 00:57:47,429 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:TASK_FINISHED]: vertexName=E077277B6A994948AFF227AB8198B4A1, taskId=task_1463533067168_0001_1_01_000000, startTime=1463533067396, finishTime=1463533067429, timeTaken=33, status=SUCCEEDED, successfulAttemptID=attempt_1463533067168_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533067419, Process_Duration=7, Process_End_Time=1463533067426, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:47,429 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:47,429 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1]
2016-05-18 00:57:47,430 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:VERTEX_FINISHED]: vertexName=E077277B6A994948AFF227AB8198B4A1, vertexId=vertex_1463533067168_0001_1_01, initRequestedTime=1463533067328, initedTime=1463533067331, startRequestedTime=1463533067338, startedTime=1463533067339, finishTime=1463533067430, timeTaken=91, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533067419, Process_Duration=7, Process_End_Time=1463533067426, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533067396, firstTasksToStart=[ task_1463533067168_0001_1_01_000000 ], lastTaskFinishTime=1463533067428, lastTasksToFinish=[ task_1463533067168_0001_1_01_000000 ], minTaskDuration=32, maxTaskDuration=32, avgTaskDuration=32.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533067168_0001_1_01_000000 ], longestDurationTasks=[ task_1463533067168_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:47,430 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:47,430 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:47,430 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:47,430 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533067168_0001_1_00 [EE29D0DC0636447AB3F136BE2A9F415F]
2016-05-18 00:57:47,431 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533067168_0001_1
2016-05-18 00:57:47,431 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533067168_0001_1
2016-05-18 00:57:47,431 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533067168_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:47,432 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 34D59C56662C445DA574320834D63DFA for vertex: vertex_1463533067168_0001_1_01 [E077277B6A994948AFF227AB8198B4A1], outputName: 34D59C56662C445DA574320834D63DFA
2016-05-18 00:57:47,432 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533067168_0001_00_000002, stoppedTime=1463533067431, exitStatus=0
2016-05-18 00:57:47,432 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533067168_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:47,439 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533067168_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:47,439 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533067168_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:47,439 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533067168_0001_00_000002, Exiting
2016-05-18 00:57:47,440 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533067168_0001_00_000002
2016-05-18 00:57:47,440 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533067168_0001_00_000002
2016-05-18 00:57:47,440 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533067168_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:47,440 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533067168_0001_00_000002 completed successfully
2016-05-18 00:57:47,440 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:34D59C56662C445DA574320834D63DFA of vertex/vertexGroup:E077277B6A994948AFF227AB8198B4A1 isVertexGroupOutput:false
2016-05-18 00:57:47,440 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:47,442 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533067168_0001_1, queueSize=0
2016-05-18 00:57:47,442 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067168_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533067168_0001_1, startTime=1463533067327, finishTime=1463533067440, timeTaken=113, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=190, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066134800, Process_Duration=9, Process_End_Time=2927066134809, Read_Duration=1, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:47,442 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533067168_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:47,442 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533067168_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:47 Completed Dag: dag_1463533067168_0001_1
2016-05-18 00:57:47,443 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533067168_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:47,443 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533067168_0001_1_post
2016-05-18 00:57:47,443 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,443 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:47,443 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:47,443 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914] every/(1/1) ...tform/ALL_'field'_RESULTS, with id=dag_1463533067168_0001_1
2016-05-18 00:57:47,443 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914] every/(1/1) ...tform/ALL_'field'_RESULTS, with id=dag_1463533067168_0001_1
2016-05-18 00:57:47,451 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...tform/ALL_'field'_RESULTS, applicationId=application_1463533067168_0001
2016-05-18 00:57:47,453 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:47,453 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:47,453 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:47,453 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:47,453 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:47,453 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:47,454 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:47,454 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:47,454 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:47,454 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:47,454 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:47,454 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:47,455 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:47,455 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:47,458 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/6C507C8347B04378BF9B44C2B693718D/E8E34CC5CAB94FCFA8819BFD948C5914/.tez/application_1463533067168_0001
2016-05-18 00:57:47,458 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_RESULTS/_temporary
2016-05-18 00:57:47,459 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.316
2016-05-18 00:57:47,533 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_'field'_RESULTS
2016-05-18 00:57:47,556 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:47,556 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:47,590 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.007
2016-05-18 00:57:47,595 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 00:57:47,595 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:47,595 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:47,595 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:47,609 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:47,614 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_ALL
2016-05-18 00:57:47,659 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:47,659 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:47,659 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_ALL"]
2016-05-18 00:57:47,659 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:47,659 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:47,660 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:47,661 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...yplatform/ALL_'field'_ALL
2016-05-18 00:57:47,661 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:47,673 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:47,677 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:47,677 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:47,678 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:47,686 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001 doesn't exist and is created
2016-05-18 00:57:47,701 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:47,701 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:47,702 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001_wd
2016-05-18 00:57:47,711 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533067677_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:47,711 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:47,711 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:47,712 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:47,712 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:47,712 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:47,713 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:47,713 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:47,714 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:47,715 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:47,715 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:47,715 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:47,716 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:47,717 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:47,717 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:47,717 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001_wd/localmode-log-dir
2016-05-18 00:57:47,717 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001_wd/localmode-log-dir/history.txt.appattempt_1463533067677_0001_000000, maxErrors=10
2016-05-18 00:57:47,717 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:47,717 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533067677_0001_000000, appSubmitTime=1463533067710, launchTime=1463533067711
2016-05-18 00:57:47,719 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:47,723 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:48032
2016-05-18 00:57:47,723 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533067677
2016-05-18 00:57:47,725 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533067677_0001_000000, startTime=1463533067725
2016-05-18 00:57:47,725 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:47,801 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:47,801 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:47,802 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...yplatform/ALL_'field'_ALL, applicationId=application_1463533067677_0001, dagName=[87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6] every/(1/1) ...yplatform/ALL_'field'_ALL
2016-05-18 00:57:47,816 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:47,816 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:47,816 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6] every/(1/1) ...yplatform/ALL_'field'_ALL
2016-05-18 00:57:47,816 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:47,817 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533067677_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001_wd/localmode-log-dir/dag_1463533067677_0001_1.dot
2016-05-18 00:57:47,818 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001_wd/localmode-log-dir/dag_1463533067677_0001_1-tez-dag.pb.txt
2016-05-18 00:57:47,824 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533067677_0001_1
2016-05-18 00:57:47,824 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,824 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6] every/(1/1) ...yplatform/ALL_'field'_ALL
2016-05-18 00:57:47 Running Dag: dag_1463533067677_0001_1
2016-05-18 00:57:47,833 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533067677_0001_1, submitTime=1463533067816
2016-05-18 00:57:47,833 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001/tez-conf.pb" } size: 36913 timestamp: 1463533067000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:47,833 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,833 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:47,834 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001/tez-conf.pb" } size: 36913 timestamp: 1463533067000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:47,834 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]
2016-05-18 00:57:47,834 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:47,835 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:47,835 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533067677_0001_1, initTime=1463533067835
2016-05-18 00:57:47,835 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533067677_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:47,835 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:47,835 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:DAG_STARTED]: dagID=dag_1463533067677_0001_1, startTime=1463533067835
2016-05-18 00:57:47,836 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533067677_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:47,836 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: AD0B1BD08A2E4D928CB6BD30829D47E1 : {00ACC10C9BC24DA59EBFB7F00C32BF3A={InputName=00ACC10C9BC24DA59EBFB7F00C32BF3A}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:47,836 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 00ACC10C9BC24DA59EBFB7F00C32BF3A, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:47,836 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,836 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,836 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,836 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,837 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...yplatform/ALL_'field'_ALL, applicationId=application_1463533067677_0001, dagName=[87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6] every/(1/1) ...yplatform/ALL_'field'_ALL
2016-05-18 00:57:47,837 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:47,837 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]
2016-05-18 00:57:47,838 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 00ACC10C9BC24DA59EBFB7F00C32BF3A on vertex vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,840 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:47,840 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]
2016-05-18 00:57:47,840 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]
2016-05-18 00:57:47,841 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533067677_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:47,841 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F], numAdditionalOutputs=1
2016-05-18 00:57:47,841 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=FEF1364544864D0FBE8365F03C21D52C, vertex=vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:47,841 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 00ACC10C9BC24DA59EBFB7F00C32BF3A asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:47,843 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:47,843 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:47,843 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D0BA5194BA6B4416816804EDF3D4667F:FEF1364544864D0FBE8365F03C21D52C using oldmapred API
2016-05-18 00:57:47,844 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=FEF1364544864D0FBE8365F03C21D52C, vertexName=D0BA5194BA6B4416816804EDF3D4667F, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:47,844 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:47,844 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:47,845 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D0BA5194BA6B4416816804EDF3D4667F, vertexId=vertex_1463533067677_0001_1_01, initRequestedTime=1463533067837, initedTime=1463533067841, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:47,845 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:47,845 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 00ACC10C9BC24DA59EBFB7F00C32BF3A on vertex vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,846 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:47,846 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {00ACC10C9BC24DA59EBFB7F00C32BF3A=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:47,846 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] parallelism set to 1
2016-05-18 00:57:47,846 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:47,847 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AD0B1BD08A2E4D928CB6BD30829D47E1, vertexId=vertex_1463533067677_0001_1_00, initRequestedTime=1463533067836, initedTime=1463533067847, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:47,847 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] with distanceFromRoot: 0
2016-05-18 00:57:47,847 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:47,847 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533067677_0001_1_00, startRequestedTime=1463533067837, startedTime=1463533067847
2016-05-18 00:57:47,847 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533067677_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:47,847 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:47,847 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533067677_0001_1_00 for vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F] numStartedSources: 1 numSources: 1
2016-05-18 00:57:47,848 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F] with distanceFromRoot: 1
2016-05-18 00:57:47,849 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533067677_0001_1_01, startRequestedTime=1463533067847, startedTime=1463533067848
2016-05-18 00:57:47,850 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:47,850 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D0BA5194BA6B4416816804EDF3D4667F with 0 source tasks and 1 pending tasks
2016-05-18 00:57:47,851 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: AD0B1BD08A2E4D928CB6BD30829D47E1 in vertex: D0BA5194BA6B4416816804EDF3D4667F numBipartiteSourceTasks: 1
2016-05-18 00:57:47,851 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in AD0B1BD08A2E4D928CB6BD30829D47E1
2016-05-18 00:57:47,851 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,852 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:TASK_STARTED]: vertexName=AD0B1BD08A2E4D928CB6BD30829D47E1, taskId=task_1463533067677_0001_1_00_000000, scheduledTime=1463533067852, launchTime=1463533067852
2016-05-18 00:57:47,852 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,853 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:47,854 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533067677_0001, with dag id: dag_1463533067677_0001_1
2016-05-18 00:57:47,854 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533067677_0001_00_000001, launchTime=1463533067854
2016-05-18 00:57:47,855 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533067677_0001_00_000001
2016-05-18 00:57:47,856 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533067677_0001_1_00_000000_0] started. Is using containerId: [container_1463533067677_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:47,856 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AD0B1BD08A2E4D928CB6BD30829D47E1, taskAttemptId=attempt_1463533067677_0001_1_00_000000_0, startTime=1463533067856, containerId=container_1463533067677_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:47,857 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533067677_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533067677_0001_1_00_000000_0
2016-05-18 00:57:47 Starting to run new task attempt: attempt_1463533067677_0001_1_00_000000_0
2016-05-18 00:57:47,858 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533067677_0001_1_00_000000_0
2016-05-18 00:57:47,858 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,858 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:47,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6] every/(1/1) ...yplatform/ALL_'field'_ALL, VertexName: AD0B1BD08A2E4D928CB6BD30829D47E1, VertexParallelism: 1, TaskAttemptID:attempt_1463533067677_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=00ACC10C9BC24DA59EBFB7F00C32BF3A, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D0BA5194BA6B4416816804EDF3D4667F, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:47,858 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:47,859 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533067677_0001_1_00_000000_0
2016-05-18 00:57:47,860 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533067677_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:47,861 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533067677_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:47,863 INFO input.MRInput (MRInput.java:initialize(440)) - 00ACC10C9BC24DA59EBFB7F00C32BF3A using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:47,863 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:47,863 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 00ACC10C9BC24DA59EBFB7F00C32BF3A
2016-05-18 00:57:47,866 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AD0B1BD08A2E4D928CB6BD30829D47E1, ordinal: 0
2016-05-18 00:57:47,866 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AD0B1BD08A2E4D928CB6BD30829D47E1, mem on start (mb), free: 184, total: 364, max: 672
2016-05-18 00:57:47,866 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:47,866 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:47,866 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:47,866 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:47,866 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[00ACC10C9BC24DA59EBFB7F00C32BF3A:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D0BA5194BA6B4416816804EDF3D4667F:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:47,866 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:47,867 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:47,867 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:47,867 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[00ACC10C9BC24DA59EBFB7F00C32BF3A:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D0BA5194BA6B4416816804EDF3D4667F:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:47,867 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:47,867 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 00ACC10C9BC24DA59EBFB7F00C32BF3A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:47,867 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:47,867 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:47,867 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 00ACC10C9BC24DA59EBFB7F00C32BF3A
2016-05-18 00:57:47,868 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:47,868 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533067677_0001_1_00_000000_0
2016-05-18 00:57:47,869 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:47,869 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 00ACC10C9BC24DA59EBFB7F00C32BF3A initialized RecordReader from event
2016-05-18 00:57:47,871 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:47,871 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 00ACC10C9BC24DA59EBFB7F00C32BF3A
2016-05-18 00:57:47,871 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 9185932CB3A249A9A59C502E5DAAFE7F
2016-05-18 00:57:47,872 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 9185932CB3A249A9A59C502E5DAAFE7F
2016-05-18 00:57:47,874 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:47,876 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:47,877 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D0BA5194BA6B4416816804EDF3D4667F using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@1529be59, comparator=cascading.tuple.hadoop.util.TupleComparator@678fcfe9, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:47,877 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D0BA5194BA6B4416816804EDF3D4667F: , UsingHashComparator=false
2016-05-18 00:57:47,890 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:47,891 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:47,891 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D0BA5194BA6B4416816804EDF3D4667F: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:47,891 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D0BA5194BA6B4416816804EDF3D4667F: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:47,891 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:47,892 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AD0B1BD08A2E4D928CB6BD30829D47E1, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:47,894 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533067677_0001_1_00_000000_0
2016-05-18 00:57:47,894 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AD0B1BD08A2E4D928CB6BD30829D47E1, mem on close (mb), free: 82, total: 364, max: 672
2016-05-18 00:57:47,894 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D0BA5194BA6B4416816804EDF3D4667F: Starting flush of map output
2016-05-18 00:57:47,895 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D0BA5194BA6B4416816804EDF3D4667F: Span0.length = 20, perItem = 6
2016-05-18 00:57:47,895 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D0BA5194BA6B4416816804EDF3D4667F: done sorting span=0, length=20, time=0
2016-05-18 00:57:47,895 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D0BA5194BA6B4416816804EDF3D4667F: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:47,896 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D0BA5194BA6B4416816804EDF3D4667F: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001_wd/localmode-local-dir/output/attempt_1463533067677_0001_1_00_000000_0_10249_0/file.out
2016-05-18 00:57:47,899 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:47 Completed running task attempt: attempt_1463533067677_0001_1_00_000000_0
2016-05-18 00:57:47,900 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533067677_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:47,900 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533067677_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:47,900 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533067677_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533067892, Process_Duration=2, Process_End_Time=1463533067894, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:47,901 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533067677_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:47,901 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:47,901 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:47,901 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,901 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AD0B1BD08A2E4D928CB6BD30829D47E1, taskAttemptId=attempt_1463533067677_0001_1_00_000000_0, creationTime=1463533067852, allocationTime=1463533067854, startTime=1463533067856, finishTime=1463533067901, timeTaken=45, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:47,901 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533067677_0001_1_00_000000_0
2016-05-18 00:57:47,901 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:TASK_FINISHED]: vertexName=AD0B1BD08A2E4D928CB6BD30829D47E1, taskId=task_1463533067677_0001_1_00_000000, startTime=1463533067856, finishTime=1463533067901, timeTaken=45, status=SUCCEEDED, successfulAttemptID=attempt_1463533067677_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533067892, Process_Duration=2, Process_End_Time=1463533067894, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:47,902 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:47,902 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,902 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:VERTEX_FINISHED]: vertexName=AD0B1BD08A2E4D928CB6BD30829D47E1, vertexId=vertex_1463533067677_0001_1_00, initRequestedTime=1463533067836, initedTime=1463533067847, startRequestedTime=1463533067837, startedTime=1463533067847, finishTime=1463533067902, timeTaken=55, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533067892, Process_Duration=2, Process_End_Time=1463533067894, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533067856, firstTasksToStart=[ task_1463533067677_0001_1_00_000000 ], lastTaskFinishTime=1463533067901, lastTasksToFinish=[ task_1463533067677_0001_1_00_000000 ], minTaskDuration=45, maxTaskDuration=45, avgTaskDuration=45.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533067677_0001_1_00_000000 ], longestDurationTasks=[ task_1463533067677_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:47,902 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:47,903 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F] attempt: attempt_1463533067677_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:47,903 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533067677_0001_00_000001
2016-05-18 00:57:47,903 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533067677_0001_00_000001, stoppedTime=1463533067903, exitStatus=0
2016-05-18 00:57:47,903 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D0BA5194BA6B4416816804EDF3D4667F
2016-05-18 00:57:47,903 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:47,903 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533067677_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:47,904 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533067677_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:47,903 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:47,903 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=AD0B1BD08A2E4D928CB6BD30829D47E1, destinationVertexName=D0BA5194BA6B4416816804EDF3D4667F. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:47,904 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]
2016-05-18 00:57:47,904 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:TASK_STARTED]: vertexName=D0BA5194BA6B4416816804EDF3D4667F, taskId=task_1463533067677_0001_1_01_000000, scheduledTime=1463533067904, launchTime=1463533067904
2016-05-18 00:57:47,904 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533067677_0001_00_000001, Exiting
2016-05-18 00:57:47,904 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533067677_0001_00_000001
2016-05-18 00:57:47,904 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]
2016-05-18 00:57:47,904 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533067677_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:47,904 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533067677_0001_00_000001
2016-05-18 00:57:47,905 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533067677_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:47,906 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533067677_0001_00_000001 completed successfully
2016-05-18 00:57:47,907 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533067677_0001_00_000002, launchTime=1463533067906
2016-05-18 00:57:47,907 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533067677_0001_00_000002
2016-05-18 00:57:47,908 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533067677_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533067677_0001_1_01_000000_0
2016-05-18 00:57:47,908 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533067677_0001_1_01_000000_0] started. Is using containerId: [container_1463533067677_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:47,908 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D0BA5194BA6B4416816804EDF3D4667F, taskAttemptId=attempt_1463533067677_0001_1_01_000000_0, startTime=1463533067908, containerId=container_1463533067677_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:47 Starting to run new task attempt: attempt_1463533067677_0001_1_01_000000_0
2016-05-18 00:57:47,909 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533067677_0001_1_01_000000_0
2016-05-18 00:57:47,909 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,910 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:47,910 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6] every/(1/1) ...yplatform/ALL_'field'_ALL, VertexName: D0BA5194BA6B4416816804EDF3D4667F, VertexParallelism: 1, TaskAttemptID:attempt_1463533067677_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AD0B1BD08A2E4D928CB6BD30829D47E1, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=FEF1364544864D0FBE8365F03C21D52C, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:47,911 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:47,911 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533067677_0001_1_01_000000_0
2016-05-18 00:57:47,912 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533067677_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]
2016-05-18 00:57:47,912 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533067677_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:47,913 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:47,915 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:47,918 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D0BA5194BA6B4416816804EDF3D4667F, ordinal: 1
2016-05-18 00:57:47,918 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D0BA5194BA6B4416816804EDF3D4667F, mem on start (mb), free: 76, total: 364, max: 672
2016-05-18 00:57:47,918 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:47,918 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:47,918 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:47,920 INFO output.MROutput (MROutput.java:initialize(421)) - FEF1364544864D0FBE8365F03C21D52C: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:47,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:47,921 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[FEF1364544864D0FBE8365F03C21D52C:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [AD0B1BD08A2E4D928CB6BD30829D47E1:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:47,921 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:47,921 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:47,921 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:47,921 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[FEF1364544864D0FBE8365F03C21D52C:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [AD0B1BD08A2E4D928CB6BD30829D47E1:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:47,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:47,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AD0B1BD08A2E4D928CB6BD30829D47E1 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:47,922 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:47,922 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:47,922 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - AD0B1BD08A2E4D928CB6BD30829D47E1: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:47,922 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - AD0B1BD08A2E4D928CB6BD30829D47E1: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:47,923 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: AD0B1BD08A2E4D928CB6BD30829D47E1 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:47,923 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AD0B1BD08A2E4D928CB6BD30829D47E1
2016-05-18 00:57:47,924 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:47,924 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533067677_0001_1_01_000000_0
2016-05-18 00:57:47,926 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AD0B1BD08A2E4D928CB6BD30829D47E1: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:47,926 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:47,927 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 9185932CB3A249A9A59C502E5DAAFE7F
2016-05-18 00:57:47,927 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_ALL"], id: FEF1364544864D0FBE8365F03C21D52C
2016-05-18 00:57:47,927 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_ALL"]
2016-05-18 00:57:47,927 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 9185932CB3A249A9A59C502E5DAAFE7F, for 1 inputs
2016-05-18 00:57:47,927 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533067677_0001_1_00_000000_0_10249} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533067927, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:47,927 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : AD0B1BD08A2E4D928CB6BD30829D47E1
2016-05-18 00:57:47,928 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:47,928 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: AD0B1BD08A2E4D928CB6BD30829D47E1, wasInterrupted=false
2016-05-18 00:57:47,928 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:47,928 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - AD0B1BD08A2E4D928CB6BD30829D47E1: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:47,928 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:47,929 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:47,929 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001_wd/localmode-local-dir/output/attempt_1463533067677_0001_1_00_000000_0_10249/file.out, len=181, isLocal=true
2016-05-18 00:57:47,929 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:47,929 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:47,930 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:47,930 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:47,930 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : AD0B1BD08A2E4D928CB6BD30829D47E1
2016-05-18 00:57:47,930 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - AD0B1BD08A2E4D928CB6BD30829D47E1: Shuffle Runner thread complete
2016-05-18 00:57:47,930 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D0BA5194BA6B4416816804EDF3D4667F, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:47,931 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:47,931 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - AD0B1BD08A2E4D928CB6BD30829D47E1: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:47,936 INFO output.MROutput (MROutput.java:close(526)) - FEF1364544864D0FBE8365F03C21D52C closed
2016-05-18 00:57:47,937 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533067677_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:47,937 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533067677_0001_1_01_000000_0
2016-05-18 00:57:47,937 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D0BA5194BA6B4416816804EDF3D4667F, mem on close (mb), free: 71, total: 364, max: 672
2016-05-18 00:57:47,937 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: AD0B1BD08A2E4D928CB6BD30829D47E1
2016-05-18 00:57:47,938 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AD0B1BD08A2E4D928CB6BD30829D47E1: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:47,938 INFO output.MROutput (MROutput.java:close(526)) - FEF1364544864D0FBE8365F03C21D52C closed
2016-05-18 00:57:47,938 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:47 Completed running task attempt: attempt_1463533067677_0001_1_01_000000_0
2016-05-18 00:57:47,938 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533067677_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:47,939 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533067677_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:47,939 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533067677_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533067930, Process_Duration=7, Process_End_Time=1463533067937, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:47,939 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533067677_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:47,939 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D0BA5194BA6B4416816804EDF3D4667F, taskAttemptId=attempt_1463533067677_0001_1_01_000000_0, creationTime=1463533067904, allocationTime=1463533067905, startTime=1463533067908, finishTime=1463533067939, timeTaken=31, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:47,940 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:47,940 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:47,940 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,940 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533067677_0001_00_000002
2016-05-18 00:57:47,940 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533067677_0001_1_01_000000_0
2016-05-18 00:57:47,941 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:TASK_FINISHED]: vertexName=D0BA5194BA6B4416816804EDF3D4667F, taskId=task_1463533067677_0001_1_01_000000, startTime=1463533067908, finishTime=1463533067940, timeTaken=32, status=SUCCEEDED, successfulAttemptID=attempt_1463533067677_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533067930, Process_Duration=7, Process_End_Time=1463533067937, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:47,941 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:47,941 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F]
2016-05-18 00:57:47,942 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:VERTEX_FINISHED]: vertexName=D0BA5194BA6B4416816804EDF3D4667F, vertexId=vertex_1463533067677_0001_1_01, initRequestedTime=1463533067837, initedTime=1463533067841, startRequestedTime=1463533067847, startedTime=1463533067848, finishTime=1463533067941, timeTaken=93, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533067930, Process_Duration=7, Process_End_Time=1463533067937, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533067908, firstTasksToStart=[ task_1463533067677_0001_1_01_000000 ], lastTaskFinishTime=1463533067939, lastTasksToFinish=[ task_1463533067677_0001_1_01_000000 ], minTaskDuration=31, maxTaskDuration=31, avgTaskDuration=31.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533067677_0001_1_01_000000 ], longestDurationTasks=[ task_1463533067677_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:47,942 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:47,942 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533067677_0001_00_000002, stoppedTime=1463533067942, exitStatus=0
2016-05-18 00:57:47,942 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:47,942 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:47,942 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533067677_0001_1_00 [AD0B1BD08A2E4D928CB6BD30829D47E1]
2016-05-18 00:57:47,953 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533067677_0001_1
2016-05-18 00:57:47,953 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533067677_0001_1
2016-05-18 00:57:47,953 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533067677_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:47,953 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533067677_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:47,953 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: FEF1364544864D0FBE8365F03C21D52C for vertex: vertex_1463533067677_0001_1_01 [D0BA5194BA6B4416816804EDF3D4667F], outputName: FEF1364544864D0FBE8365F03C21D52C
2016-05-18 00:57:47,954 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533067677_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:47,954 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533067677_0001_00_000002: 14 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:47,954 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533067677_0001_00_000002, Exiting
2016-05-18 00:57:47,954 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533067677_0001_00_000002
2016-05-18 00:57:47,954 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533067677_0001_00_000002
2016-05-18 00:57:47,954 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533067677_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:47,954 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533067677_0001_00_000002 completed successfully
2016-05-18 00:57:47,960 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:FEF1364544864D0FBE8365F03C21D52C of vertex/vertexGroup:D0BA5194BA6B4416816804EDF3D4667F isVertexGroupOutput:false
2016-05-18 00:57:47,960 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:47,961 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533067677_0001_1, queueSize=0
2016-05-18 00:57:47,961 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533067677_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533067677_0001_1, startTime=1463533067835, finishTime=1463533067960, timeTaken=125, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=230, AM_GC_TIME_MILLIS=10, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066135822, Process_Duration=9, Process_End_Time=2927066135831, Read_Duration=1, Tuples_Read=40, Tuples_Written=40, Write_Duration=3, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:47,961 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533067677_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:47,961 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533067677_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:47 Completed Dag: dag_1463533067677_0001_1
2016-05-18 00:57:47,962 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533067677_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:47,962 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533067677_0001_1_post
2016-05-18 00:57:47,962 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:47,962 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:47,962 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:47,962 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6] every/(1/1) ...yplatform/ALL_'field'_ALL, with id=dag_1463533067677_0001_1
2016-05-18 00:57:47,962 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6] every/(1/1) ...yplatform/ALL_'field'_ALL, with id=dag_1463533067677_0001_1
2016-05-18 00:57:47,970 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...yplatform/ALL_'field'_ALL, applicationId=application_1463533067677_0001
2016-05-18 00:57:47,972 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:47,972 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:47,972 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:47,972 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:47,972 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:47,973 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:47,973 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:47,973 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:47,973 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:47,974 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:47,974 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:47,974 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:47,975 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:47,976 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:47,978 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87BD477A02014BD18B7E3E16293AF6A6/D70C65EA481849A8842FBB51CEE0F9A6/.tez/application_1463533067677_0001
2016-05-18 00:57:47,978 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_ALL/_temporary
2016-05-18 00:57:47,979 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.319
2016-05-18 00:57:48,043 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_'field'_ALL
2016-05-18 00:57:48,065 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:48,065 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:48,094 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.003
2016-05-18 00:57:48,095 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.004
2016-05-18 00:57:48,095 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:48,096 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:48,096 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_'field'_REPLACE
2016-05-18 00:57:48,119 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:48,119 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:48,153 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.010
2016-05-18 00:57:48,158 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 00:57:48,158 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:48,158 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:48,158 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:48,173 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:48,177 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_SWAP
2016-05-18 00:57:48,220 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:48,220 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:48,220 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_SWAP"]
2016-05-18 00:57:48,220 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:48,221 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:48,221 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:48,221 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...platform/ALL_'field'_SWAP
2016-05-18 00:57:48,221 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:48,235 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:48,239 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:48,239 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:48,239 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:48,248 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001 doesn't exist and is created
2016-05-18 00:57:48,261 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:48,262 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:48,262 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001_wd
2016-05-18 00:57:48,271 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:48,271 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:48,271 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533068239_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:48,272 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:48,272 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:48,273 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:48,273 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:48,273 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:48,273 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:48,274 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:48,275 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:48,275 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:48,275 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:48,275 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:48,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:48,276 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:48,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:48,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:48,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:48,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:48,276 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:48,276 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:48,278 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:48,279 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:48,279 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:48,279 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001_wd/localmode-log-dir
2016-05-18 00:57:48,279 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001_wd/localmode-log-dir/history.txt.appattempt_1463533068239_0001_000000, maxErrors=10
2016-05-18 00:57:48,280 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:48,280 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533068239_0001_000000, appSubmitTime=1463533068271, launchTime=1463533068271
2016-05-18 00:57:48,283 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:48,284 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:48279
2016-05-18 00:57:48,285 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533068239
2016-05-18 00:57:48,287 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533068239_0001_000000, startTime=1463533068286
2016-05-18 00:57:48,287 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:48,362 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:48,362 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:48,362 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...platform/ALL_'field'_SWAP, applicationId=application_1463533068239_0001, dagName=[1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE] every/(1/1) ...platform/ALL_'field'_SWAP
2016-05-18 00:57:48,378 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE] every/(1/1) ...platform/ALL_'field'_SWAP
2016-05-18 00:57:48,379 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:48,379 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533068239_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001_wd/localmode-log-dir/dag_1463533068239_0001_1.dot
2016-05-18 00:57:48,380 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001_wd/localmode-log-dir/dag_1463533068239_0001_1-tez-dag.pb.txt
2016-05-18 00:57:48,385 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533068239_0001_1
2016-05-18 00:57:48,385 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:48,385 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE] every/(1/1) ...platform/ALL_'field'_SWAP
2016-05-18 00:57:48 Running Dag: dag_1463533068239_0001_1
2016-05-18 00:57:48,394 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533068239_0001_1, submitTime=1463533068378
2016-05-18 00:57:48,394 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001/tez-conf.pb" } size: 36913 timestamp: 1463533068000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:48,396 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,396 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:48,396 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001/tez-conf.pb" } size: 36913 timestamp: 1463533068000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:48,397 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]
2016-05-18 00:57:48,397 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:48,397 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:48,398 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533068239_0001_1, initTime=1463533068397
2016-05-18 00:57:48,398 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533068239_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:48,398 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:48,398 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:DAG_STARTED]: dagID=dag_1463533068239_0001_1, startTime=1463533068398
2016-05-18 00:57:48,398 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533068239_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:48,399 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: DE817EBA21044BBE9008FF9864FD8059 : {70C2906A700749A8844FE7224409CC1F={InputName=70C2906A700749A8844FE7224409CC1F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:48,399 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 70C2906A700749A8844FE7224409CC1F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:48,399 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,399 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...platform/ALL_'field'_SWAP, applicationId=application_1463533068239_0001, dagName=[1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE] every/(1/1) ...platform/ALL_'field'_SWAP
2016-05-18 00:57:48,399 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,400 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,400 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,401 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:48,401 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]
2016-05-18 00:57:48,402 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 70C2906A700749A8844FE7224409CC1F on vertex vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,404 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:48,404 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]
2016-05-18 00:57:48,404 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]
2016-05-18 00:57:48,404 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 70C2906A700749A8844FE7224409CC1F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:48,404 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533068239_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:48,405 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D], numAdditionalOutputs=1
2016-05-18 00:57:48,405 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=9AEBE54703804F5EA68E47F517262212, vertex=vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:48,406 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:48,407 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:48,407 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for C78BD280B23443AE98B20C42D0653A1D:9AEBE54703804F5EA68E47F517262212 using oldmapred API
2016-05-18 00:57:48,407 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=9AEBE54703804F5EA68E47F517262212, vertexName=C78BD280B23443AE98B20C42D0653A1D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:48,407 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:48,408 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:48,408 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 70C2906A700749A8844FE7224409CC1F on vertex vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,408 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C78BD280B23443AE98B20C42D0653A1D, vertexId=vertex_1463533068239_0001_1_01, initRequestedTime=1463533068401, initedTime=1463533068405, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:48,409 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:48,409 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:48,409 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {70C2906A700749A8844FE7224409CC1F=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:48,410 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] parallelism set to 1
2016-05-18 00:57:48,410 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:48,410 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DE817EBA21044BBE9008FF9864FD8059, vertexId=vertex_1463533068239_0001_1_00, initRequestedTime=1463533068398, initedTime=1463533068410, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:48,411 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] with distanceFromRoot: 0
2016-05-18 00:57:48,411 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:48,411 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533068239_0001_1_00, startRequestedTime=1463533068401, startedTime=1463533068411
2016-05-18 00:57:48,412 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533068239_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:48,412 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:48,412 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533068239_0001_1_00 for vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D] numStartedSources: 1 numSources: 1
2016-05-18 00:57:48,412 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D] with distanceFromRoot: 1
2016-05-18 00:57:48,412 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in DE817EBA21044BBE9008FF9864FD8059
2016-05-18 00:57:48,413 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533068239_0001_1_01, startRequestedTime=1463533068412, startedTime=1463533068413
2016-05-18 00:57:48,413 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:48,413 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: C78BD280B23443AE98B20C42D0653A1D with 0 source tasks and 1 pending tasks
2016-05-18 00:57:48,414 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,414 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: DE817EBA21044BBE9008FF9864FD8059 in vertex: C78BD280B23443AE98B20C42D0653A1D numBipartiteSourceTasks: 1
2016-05-18 00:57:48,414 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:TASK_STARTED]: vertexName=DE817EBA21044BBE9008FF9864FD8059, taskId=task_1463533068239_0001_1_00_000000, scheduledTime=1463533068414, launchTime=1463533068414
2016-05-18 00:57:48,415 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,416 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:48,417 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533068239_0001_00_000001, launchTime=1463533068417
2016-05-18 00:57:48,417 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533068239_0001_00_000001
2016-05-18 00:57:48,418 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533068239_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533068239_0001_1_00_000000_0
2016-05-18 00:57:48,418 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533068239_0001_1_00_000000_0] started. Is using containerId: [container_1463533068239_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:48,418 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DE817EBA21044BBE9008FF9864FD8059, taskAttemptId=attempt_1463533068239_0001_1_00_000000_0, startTime=1463533068418, containerId=container_1463533068239_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:48,418 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533068239_0001, with dag id: dag_1463533068239_0001_1
2016-05-18 00:57:48 Starting to run new task attempt: attempt_1463533068239_0001_1_00_000000_0
2016-05-18 00:57:48,418 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533068239_0001_1_00_000000_0
2016-05-18 00:57:48,419 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:48,419 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:48,419 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE] every/(1/1) ...platform/ALL_'field'_SWAP, VertexName: DE817EBA21044BBE9008FF9864FD8059, VertexParallelism: 1, TaskAttemptID:attempt_1463533068239_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=70C2906A700749A8844FE7224409CC1F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C78BD280B23443AE98B20C42D0653A1D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:48,419 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:48,421 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533068239_0001_1_00_000000_0
2016-05-18 00:57:48,422 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533068239_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:48,422 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533068239_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:48,424 INFO input.MRInput (MRInput.java:initialize(440)) - 70C2906A700749A8844FE7224409CC1F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:48,424 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 70C2906A700749A8844FE7224409CC1F
2016-05-18 00:57:48,424 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:48,428 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DE817EBA21044BBE9008FF9864FD8059, ordinal: 0
2016-05-18 00:57:48,428 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DE817EBA21044BBE9008FF9864FD8059, mem on start (mb), free: 150, total: 364, max: 672
2016-05-18 00:57:48,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:48,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:48,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:48,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:48,428 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[70C2906A700749A8844FE7224409CC1F:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [C78BD280B23443AE98B20C42D0653A1D:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:48,428 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:48,428 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:48,429 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:48,429 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[70C2906A700749A8844FE7224409CC1F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [C78BD280B23443AE98B20C42D0653A1D:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:48,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:48,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 70C2906A700749A8844FE7224409CC1F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:48,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:48,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:48,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 70C2906A700749A8844FE7224409CC1F
2016-05-18 00:57:48,429 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:48,429 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533068239_0001_1_00_000000_0
2016-05-18 00:57:48,431 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:48,431 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 70C2906A700749A8844FE7224409CC1F initialized RecordReader from event
2016-05-18 00:57:48,432 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:48,432 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 70C2906A700749A8844FE7224409CC1F
2016-05-18 00:57:48,432 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 43228B0D759B41448DE2AAA44D756AC6
2016-05-18 00:57:48,432 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 43228B0D759B41448DE2AAA44D756AC6
2016-05-18 00:57:48,433 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:48,435 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:48,435 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - C78BD280B23443AE98B20C42D0653A1D using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@66fa74b9, comparator=cascading.tuple.hadoop.util.TupleComparator@45055280, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:48,435 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for C78BD280B23443AE98B20C42D0653A1D: , UsingHashComparator=false
2016-05-18 00:57:48,462 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:48,463 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:48,463 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for C78BD280B23443AE98B20C42D0653A1D: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:48,463 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - C78BD280B23443AE98B20C42D0653A1D: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:48,463 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:48,463 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DE817EBA21044BBE9008FF9864FD8059, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:48,465 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533068239_0001_1_00_000000_0
2016-05-18 00:57:48,465 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DE817EBA21044BBE9008FF9864FD8059, mem on close (mb), free: 177, total: 364, max: 672
2016-05-18 00:57:48,465 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - C78BD280B23443AE98B20C42D0653A1D: Starting flush of map output
2016-05-18 00:57:48,465 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - C78BD280B23443AE98B20C42D0653A1D: Span0.length = 20, perItem = 6
2016-05-18 00:57:48,466 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - C78BD280B23443AE98B20C42D0653A1D: done sorting span=0, length=20, time=0
2016-05-18 00:57:48,466 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - C78BD280B23443AE98B20C42D0653A1D: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:48,467 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - C78BD280B23443AE98B20C42D0653A1D: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001_wd/localmode-local-dir/output/attempt_1463533068239_0001_1_00_000000_0_10255_0/file.out
2016-05-18 00:57:48,470 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:48 Completed running task attempt: attempt_1463533068239_0001_1_00_000000_0
2016-05-18 00:57:48,470 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533068239_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:48,470 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533068239_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:48,471 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533068239_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533068463, Process_Duration=2, Process_End_Time=1463533068465, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:48,471 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533068239_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:48,471 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:48,471 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DE817EBA21044BBE9008FF9864FD8059, taskAttemptId=attempt_1463533068239_0001_1_00_000000_0, creationTime=1463533068415, allocationTime=1463533068416, startTime=1463533068418, finishTime=1463533068471, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:48,471 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:48,472 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:48,472 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533068239_0001_1_00_000000_0
2016-05-18 00:57:48,472 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533068239_0001_00_000001
2016-05-18 00:57:48,472 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:TASK_FINISHED]: vertexName=DE817EBA21044BBE9008FF9864FD8059, taskId=task_1463533068239_0001_1_00_000000, startTime=1463533068418, finishTime=1463533068472, timeTaken=54, status=SUCCEEDED, successfulAttemptID=attempt_1463533068239_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533068463, Process_Duration=2, Process_End_Time=1463533068465, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:48,472 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:48,472 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,473 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:VERTEX_FINISHED]: vertexName=DE817EBA21044BBE9008FF9864FD8059, vertexId=vertex_1463533068239_0001_1_00, initRequestedTime=1463533068398, initedTime=1463533068410, startRequestedTime=1463533068401, startedTime=1463533068411, finishTime=1463533068473, timeTaken=62, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533068463, Process_Duration=2, Process_End_Time=1463533068465, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533068418, firstTasksToStart=[ task_1463533068239_0001_1_00_000000 ], lastTaskFinishTime=1463533068471, lastTasksToFinish=[ task_1463533068239_0001_1_00_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533068239_0001_1_00_000000 ], longestDurationTasks=[ task_1463533068239_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:48,473 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:48,473 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D] attempt: attempt_1463533068239_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:48,473 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C78BD280B23443AE98B20C42D0653A1D
2016-05-18 00:57:48,474 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533068239_0001_00_000001, stoppedTime=1463533068473, exitStatus=0
2016-05-18 00:57:48,474 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=DE817EBA21044BBE9008FF9864FD8059, destinationVertexName=C78BD280B23443AE98B20C42D0653A1D. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:48,474 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]
2016-05-18 00:57:48,474 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:48,474 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:48,474 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533068239_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:48,474 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:TASK_STARTED]: vertexName=C78BD280B23443AE98B20C42D0653A1D, taskId=task_1463533068239_0001_1_01_000000, scheduledTime=1463533068474, launchTime=1463533068474
2016-05-18 00:57:48,474 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]
2016-05-18 00:57:48,476 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533068239_0001_00_000002, launchTime=1463533068476
2016-05-18 00:57:48,476 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533068239_0001_00_000002
2016-05-18 00:57:48,477 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533068239_0001_1_01_000000_0] started. Is using containerId: [container_1463533068239_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:48,477 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C78BD280B23443AE98B20C42D0653A1D, taskAttemptId=attempt_1463533068239_0001_1_01_000000_0, startTime=1463533068476, containerId=container_1463533068239_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:48,477 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533068239_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533068239_0001_1_01_000000_0
2016-05-18 00:57:48 Starting to run new task attempt: attempt_1463533068239_0001_1_01_000000_0
2016-05-18 00:57:48,478 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533068239_0001_1_01_000000_0
2016-05-18 00:57:48,478 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:48,478 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:48,478 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE] every/(1/1) ...platform/ALL_'field'_SWAP, VertexName: C78BD280B23443AE98B20C42D0653A1D, VertexParallelism: 1, TaskAttemptID:attempt_1463533068239_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DE817EBA21044BBE9008FF9864FD8059, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=9AEBE54703804F5EA68E47F517262212, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:48,479 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:48,479 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533068239_0001_1_01_000000_0
2016-05-18 00:57:48,480 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533068239_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]
2016-05-18 00:57:48,480 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533068239_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:48,481 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:48,482 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:48,483 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533068239_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:48,483 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533068239_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:48,483 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533068239_0001_00_000001, Exiting
2016-05-18 00:57:48,483 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533068239_0001_00_000001
2016-05-18 00:57:48,483 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533068239_0001_00_000001
2016-05-18 00:57:48,484 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533068239_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:48,484 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533068239_0001_00_000001 completed successfully
2016-05-18 00:57:48,486 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C78BD280B23443AE98B20C42D0653A1D, ordinal: 1
2016-05-18 00:57:48,486 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C78BD280B23443AE98B20C42D0653A1D, mem on start (mb), free: 171, total: 364, max: 672
2016-05-18 00:57:48,487 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:48,487 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:48,487 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:48,493 INFO output.MROutput (MROutput.java:initialize(421)) - 9AEBE54703804F5EA68E47F517262212: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:48,493 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:48,493 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DE817EBA21044BBE9008FF9864FD8059:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [9AEBE54703804F5EA68E47F517262212:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:48,493 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:48,493 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:48,493 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:48,493 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DE817EBA21044BBE9008FF9864FD8059:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [9AEBE54703804F5EA68E47F517262212:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:48,494 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:48,494 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: DE817EBA21044BBE9008FF9864FD8059 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:48,494 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:48,494 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:48,494 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - DE817EBA21044BBE9008FF9864FD8059: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:48,494 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - DE817EBA21044BBE9008FF9864FD8059: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:48,495 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: DE817EBA21044BBE9008FF9864FD8059 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:48,496 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: DE817EBA21044BBE9008FF9864FD8059
2016-05-18 00:57:48,496 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:48,496 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533068239_0001_1_01_000000_0
2016-05-18 00:57:48,498 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DE817EBA21044BBE9008FF9864FD8059: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:48,499 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:48,500 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 43228B0D759B41448DE2AAA44D756AC6
2016-05-18 00:57:48,500 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_SWAP"], id: 9AEBE54703804F5EA68E47F517262212
2016-05-18 00:57:48,500 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_SWAP"]
2016-05-18 00:57:48,500 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 43228B0D759B41448DE2AAA44D756AC6, for 1 inputs
2016-05-18 00:57:48,502 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533068239_0001_1_00_000000_0_10255} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533068502, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:48,502 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : DE817EBA21044BBE9008FF9864FD8059
2016-05-18 00:57:48,502 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:48,503 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: DE817EBA21044BBE9008FF9864FD8059, wasInterrupted=false
2016-05-18 00:57:48,503 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:48,503 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:48,504 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:48,504 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001_wd/localmode-local-dir/output/attempt_1463533068239_0001_1_00_000000_0_10255/file.out, len=181, isLocal=true
2016-05-18 00:57:48,504 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:48,504 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:48,505 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:48,505 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:48,505 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : DE817EBA21044BBE9008FF9864FD8059
2016-05-18 00:57:48,505 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - DE817EBA21044BBE9008FF9864FD8059: Shuffle Runner thread complete
2016-05-18 00:57:48,505 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C78BD280B23443AE98B20C42D0653A1D, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:48,505 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - DE817EBA21044BBE9008FF9864FD8059: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:48,507 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:48,507 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - DE817EBA21044BBE9008FF9864FD8059: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:48,513 INFO output.MROutput (MROutput.java:close(526)) - 9AEBE54703804F5EA68E47F517262212 closed
2016-05-18 00:57:48,513 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533068239_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:48,514 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533068239_0001_1_01_000000_0
2016-05-18 00:57:48,514 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C78BD280B23443AE98B20C42D0653A1D, mem on close (mb), free: 168, total: 364, max: 672
2016-05-18 00:57:48,515 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: DE817EBA21044BBE9008FF9864FD8059
2016-05-18 00:57:48,515 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - DE817EBA21044BBE9008FF9864FD8059: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:48,515 INFO output.MROutput (MROutput.java:close(526)) - 9AEBE54703804F5EA68E47F517262212 closed
2016-05-18 00:57:48,516 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:48 Completed running task attempt: attempt_1463533068239_0001_1_01_000000_0
2016-05-18 00:57:48,517 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533068239_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:48,517 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533068239_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:48,518 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533068239_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533068505, Process_Duration=9, Process_End_Time=1463533068514, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:48,518 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533068239_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:48,518 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:48,518 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:48,519 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:48,519 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C78BD280B23443AE98B20C42D0653A1D, taskAttemptId=attempt_1463533068239_0001_1_01_000000_0, creationTime=1463533068474, allocationTime=1463533068475, startTime=1463533068476, finishTime=1463533068519, timeTaken=43, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:48,519 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533068239_0001_1_01_000000_0
2016-05-18 00:57:48,519 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:TASK_FINISHED]: vertexName=C78BD280B23443AE98B20C42D0653A1D, taskId=task_1463533068239_0001_1_01_000000, startTime=1463533068476, finishTime=1463533068519, timeTaken=43, status=SUCCEEDED, successfulAttemptID=attempt_1463533068239_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533068505, Process_Duration=9, Process_End_Time=1463533068514, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:48,519 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533068239_0001_00_000002
2016-05-18 00:57:48,520 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:48,520 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D]
2016-05-18 00:57:48,520 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:VERTEX_FINISHED]: vertexName=C78BD280B23443AE98B20C42D0653A1D, vertexId=vertex_1463533068239_0001_1_01, initRequestedTime=1463533068401, initedTime=1463533068405, startRequestedTime=1463533068412, startedTime=1463533068413, finishTime=1463533068520, timeTaken=107, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533068505, Process_Duration=9, Process_End_Time=1463533068514, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533068476, firstTasksToStart=[ task_1463533068239_0001_1_01_000000 ], lastTaskFinishTime=1463533068519, lastTasksToFinish=[ task_1463533068239_0001_1_01_000000 ], minTaskDuration=43, maxTaskDuration=43, avgTaskDuration=43.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533068239_0001_1_01_000000 ], longestDurationTasks=[ task_1463533068239_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:48,520 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:48,520 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533068239_0001_00_000002, stoppedTime=1463533068520, exitStatus=0
2016-05-18 00:57:48,521 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:48,521 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:48,521 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533068239_0001_1_00 [DE817EBA21044BBE9008FF9864FD8059]
2016-05-18 00:57:48,521 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533068239_0001_1
2016-05-18 00:57:48,521 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533068239_0001_1
2016-05-18 00:57:48,521 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533068239_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:48,521 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 9AEBE54703804F5EA68E47F517262212 for vertex: vertex_1463533068239_0001_1_01 [C78BD280B23443AE98B20C42D0653A1D], outputName: 9AEBE54703804F5EA68E47F517262212
2016-05-18 00:57:48,521 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533068239_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:48,528 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:9AEBE54703804F5EA68E47F517262212 of vertex/vertexGroup:C78BD280B23443AE98B20C42D0653A1D isVertexGroupOutput:false
2016-05-18 00:57:48,528 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:48,529 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533068239_0001_1, queueSize=0
2016-05-18 00:57:48,529 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068239_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533068239_0001_1, startTime=1463533068398, finishTime=1463533068528, timeTaken=130, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=290, AM_GC_TIME_MILLIS=14, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=11, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066136968, Process_Duration=11, Process_End_Time=2927066136979, Read_Duration=2, Tuples_Read=40, Tuples_Written=40, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:48,529 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533068239_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:48,530 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533068239_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:48 Completed Dag: dag_1463533068239_0001_1
2016-05-18 00:57:48,530 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533068239_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:48,530 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533068239_0001_1_post
2016-05-18 00:57:48,530 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:48,530 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:48,530 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:48,530 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE] every/(1/1) ...platform/ALL_'field'_SWAP, with id=dag_1463533068239_0001_1
2016-05-18 00:57:48,530 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE] every/(1/1) ...platform/ALL_'field'_SWAP, with id=dag_1463533068239_0001_1
2016-05-18 00:57:48,531 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533068239_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:48,531 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533068239_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:48,531 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533068239_0001_00_000002, Exiting
2016-05-18 00:57:48,531 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533068239_0001_00_000002
2016-05-18 00:57:48,531 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533068239_0001_00_000002
2016-05-18 00:57:48,531 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533068239_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:48,531 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533068239_0001_00_000002 completed successfully
2016-05-18 00:57:48,542 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...platform/ALL_'field'_SWAP, applicationId=application_1463533068239_0001
2016-05-18 00:57:48,544 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:48,544 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:48,544 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:48,544 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:48,544 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:48,545 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:48,545 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:48,545 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:48,545 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:48,545 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:48,545 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:48,546 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:48,546 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:48,547 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:48,550 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/1663DED68C8E4A5E8D8B6AFDACEC0721/AD933B8D5EB64694A91D9B9E348AD2FE/.tez/application_1463533068239_0001
2016-05-18 00:57:48,551 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_'field'_SWAP/_temporary
2016-05-18 00:57:48,551 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.330
2016-05-18 00:57:48,621 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_'field'_SWAP
2016-05-18 00:57:48,643 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:48,643 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:48,673 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.005
2016-05-18 00:57:48,683 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.014
2016-05-18 00:57:48,683 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:48,684 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:48,684 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:48,699 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:48,704 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_-1
2016-05-18 00:57:48,728 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:48,728 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:48,748 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:48,749 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:48,749 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_-1"]
2016-05-18 00:57:48,749 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:48,749 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:48,749 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:48,749 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/ALL_UNKNOWN_-1
2016-05-18 00:57:48,750 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:48,761 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:48,765 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:48,766 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:48,766 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:48,775 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001 doesn't exist and is created
2016-05-18 00:57:48,789 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:48,789 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:48,790 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001_wd
2016-05-18 00:57:48,799 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533068766_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:48,799 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:48,799 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:48,800 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:48,800 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:48,800 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:48,801 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:48,801 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:48,802 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:48,802 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:48,802 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:48,802 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:48,803 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:48,803 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:48,803 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:48,803 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:48,803 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:48,803 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:48,803 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:48,803 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:48,804 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:48,805 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:48,805 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:48,805 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001_wd/localmode-log-dir
2016-05-18 00:57:48,805 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001_wd/localmode-log-dir/history.txt.appattempt_1463533068766_0001_000000, maxErrors=10
2016-05-18 00:57:48,806 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:48,806 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533068766_0001_000000, appSubmitTime=1463533068798, launchTime=1463533068798
2016-05-18 00:57:48,810 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:48,812 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:47015
2016-05-18 00:57:48,813 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533068766
2016-05-18 00:57:48,814 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533068766_0001_000000, startTime=1463533068814
2016-05-18 00:57:48,814 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:48,889 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:48,890 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:48,890 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/ALL_UNKNOWN_-1, applicationId=application_1463533068766_0001, dagName=[298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC] every/(1/1) ...lyplatform/ALL_UNKNOWN_-1
2016-05-18 00:57:48,904 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC] every/(1/1) ...lyplatform/ALL_UNKNOWN_-1
2016-05-18 00:57:48,904 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:48,904 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533068766_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001_wd/localmode-log-dir/dag_1463533068766_0001_1.dot
2016-05-18 00:57:48,905 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001_wd/localmode-log-dir/dag_1463533068766_0001_1-tez-dag.pb.txt
2016-05-18 00:57:48,910 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533068766_0001_1
2016-05-18 00:57:48,911 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:48,911 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC] every/(1/1) ...lyplatform/ALL_UNKNOWN_-1
2016-05-18 00:57:48 Running Dag: dag_1463533068766_0001_1
2016-05-18 00:57:48,920 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533068766_0001_1, submitTime=1463533068904
2016-05-18 00:57:48,921 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001/tez-conf.pb" } size: 36913 timestamp: 1463533068000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:48,921 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,921 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:48,921 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001/tez-conf.pb" } size: 36913 timestamp: 1463533068000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:48,921 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]
2016-05-18 00:57:48,921 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:48,922 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:48,922 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533068766_0001_1, initTime=1463533068922
2016-05-18 00:57:48,922 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533068766_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:48,922 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:48,922 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:DAG_STARTED]: dagID=dag_1463533068766_0001_1, startTime=1463533068922
2016-05-18 00:57:48,922 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533068766_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:48,922 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: CB171CD582704EB9984809D454ED3C83 : {6FC0E6088D5249BBB3FC9D3201D0E0DC={InputName=6FC0E6088D5249BBB3FC9D3201D0E0DC}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:48,922 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 6FC0E6088D5249BBB3FC9D3201D0E0DC, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:48,922 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,923 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,923 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,923 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,924 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/ALL_UNKNOWN_-1, applicationId=application_1463533068766_0001, dagName=[298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC] every/(1/1) ...lyplatform/ALL_UNKNOWN_-1
2016-05-18 00:57:48,924 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:48,924 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]
2016-05-18 00:57:48,925 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 6FC0E6088D5249BBB3FC9D3201D0E0DC on vertex vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,927 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 6FC0E6088D5249BBB3FC9D3201D0E0DC asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:48,928 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:48,928 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]
2016-05-18 00:57:48,928 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]
2016-05-18 00:57:48,929 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533068766_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:48,929 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43], numAdditionalOutputs=1
2016-05-18 00:57:48,929 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=AFD8CA50DE054A78997B3E905D26F519, vertex=vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:48,930 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:48,930 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:48,930 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:48,930 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:48,931 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DF9ED871BF7E4AC0A80676380D413F43:AFD8CA50DE054A78997B3E905D26F519 using oldmapred API
2016-05-18 00:57:48,931 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=AFD8CA50DE054A78997B3E905D26F519, vertexName=DF9ED871BF7E4AC0A80676380D413F43, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:48,931 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 6FC0E6088D5249BBB3FC9D3201D0E0DC on vertex vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,932 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DF9ED871BF7E4AC0A80676380D413F43, vertexId=vertex_1463533068766_0001_1_01, initRequestedTime=1463533068924, initedTime=1463533068929, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:48,932 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:48,933 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:48,933 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {6FC0E6088D5249BBB3FC9D3201D0E0DC=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:48,933 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] parallelism set to 1
2016-05-18 00:57:48,933 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:48,934 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CB171CD582704EB9984809D454ED3C83, vertexId=vertex_1463533068766_0001_1_00, initRequestedTime=1463533068922, initedTime=1463533068934, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:48,934 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] with distanceFromRoot: 0
2016-05-18 00:57:48,934 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:48,934 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533068766_0001_1_00, startRequestedTime=1463533068924, startedTime=1463533068934
2016-05-18 00:57:48,934 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533068766_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:48,934 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:48,935 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533068766_0001_1_00 for vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43] numStartedSources: 1 numSources: 1
2016-05-18 00:57:48,935 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43] with distanceFromRoot: 1
2016-05-18 00:57:48,935 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533068766_0001_1_01, startRequestedTime=1463533068934, startedTime=1463533068935
2016-05-18 00:57:48,935 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:48,936 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DF9ED871BF7E4AC0A80676380D413F43 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:48,936 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: CB171CD582704EB9984809D454ED3C83 in vertex: DF9ED871BF7E4AC0A80676380D413F43 numBipartiteSourceTasks: 1
2016-05-18 00:57:48,937 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in CB171CD582704EB9984809D454ED3C83
2016-05-18 00:57:48,937 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,937 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:TASK_STARTED]: vertexName=CB171CD582704EB9984809D454ED3C83, taskId=task_1463533068766_0001_1_00_000000, scheduledTime=1463533068937, launchTime=1463533068937
2016-05-18 00:57:48,937 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:48,938 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:48,940 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533068766_0001_00_000001, launchTime=1463533068939
2016-05-18 00:57:48,940 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533068766_0001_00_000001
2016-05-18 00:57:48,941 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533068766_0001_1_00_000000_0] started. Is using containerId: [container_1463533068766_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:48,941 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CB171CD582704EB9984809D454ED3C83, taskAttemptId=attempt_1463533068766_0001_1_00_000000_0, startTime=1463533068940, containerId=container_1463533068766_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:48,941 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533068766_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533068766_0001_1_00_000000_0
2016-05-18 00:57:48 Starting to run new task attempt: attempt_1463533068766_0001_1_00_000000_0
2016-05-18 00:57:48,942 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533068766_0001_1_00_000000_0
2016-05-18 00:57:48,942 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:48,942 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533068766_0001, with dag id: dag_1463533068766_0001_1
2016-05-18 00:57:48,942 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:48,942 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC] every/(1/1) ...lyplatform/ALL_UNKNOWN_-1, VertexName: CB171CD582704EB9984809D454ED3C83, VertexParallelism: 1, TaskAttemptID:attempt_1463533068766_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=6FC0E6088D5249BBB3FC9D3201D0E0DC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DF9ED871BF7E4AC0A80676380D413F43, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:48,942 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:48,943 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533068766_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:48,944 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533068766_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:48,943 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533068766_0001_1_00_000000_0
2016-05-18 00:57:48,959 INFO input.MRInput (MRInput.java:initialize(440)) - 6FC0E6088D5249BBB3FC9D3201D0E0DC using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:48,959 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 6FC0E6088D5249BBB3FC9D3201D0E0DC
2016-05-18 00:57:48,960 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:48,963 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CB171CD582704EB9984809D454ED3C83, ordinal: 0
2016-05-18 00:57:48,963 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CB171CD582704EB9984809D454ED3C83, mem on start (mb), free: 278, total: 367, max: 672
2016-05-18 00:57:48,964 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:48,964 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:48,964 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:48,964 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:48,964 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6FC0E6088D5249BBB3FC9D3201D0E0DC:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [DF9ED871BF7E4AC0A80676380D413F43:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:48,964 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:48,965 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:48,965 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:48,965 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6FC0E6088D5249BBB3FC9D3201D0E0DC:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [DF9ED871BF7E4AC0A80676380D413F43:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:48,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:48,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 6FC0E6088D5249BBB3FC9D3201D0E0DC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:48,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:48,966 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:48,965 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 6FC0E6088D5249BBB3FC9D3201D0E0DC
2016-05-18 00:57:48,966 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:48,966 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533068766_0001_1_00_000000_0
2016-05-18 00:57:48,967 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:48,968 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 6FC0E6088D5249BBB3FC9D3201D0E0DC initialized RecordReader from event
2016-05-18 00:57:48,970 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:48,970 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 6FC0E6088D5249BBB3FC9D3201D0E0DC
2016-05-18 00:57:48,971 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: B969D98C6BEB4BF388C5EF1E647E1F00
2016-05-18 00:57:48,971 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] B969D98C6BEB4BF388C5EF1E647E1F00
2016-05-18 00:57:48,972 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:48,975 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:48,975 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DF9ED871BF7E4AC0A80676380D413F43 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@44f1e8f0, comparator=cascading.tuple.hadoop.util.TupleComparator@65f407ec, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:48,975 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DF9ED871BF7E4AC0A80676380D413F43: , UsingHashComparator=false
2016-05-18 00:57:48,987 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:48,988 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:48,988 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DF9ED871BF7E4AC0A80676380D413F43: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:48,988 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DF9ED871BF7E4AC0A80676380D413F43: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:48,988 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:48,989 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CB171CD582704EB9984809D454ED3C83, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:57:48,991 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533068766_0001_1_00_000000_0
2016-05-18 00:57:48,991 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CB171CD582704EB9984809D454ED3C83, mem on close (mb), free: 176, total: 367, max: 672
2016-05-18 00:57:48,991 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DF9ED871BF7E4AC0A80676380D413F43: Starting flush of map output
2016-05-18 00:57:48,991 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DF9ED871BF7E4AC0A80676380D413F43: Span0.length = 20, perItem = 6
2016-05-18 00:57:48,992 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DF9ED871BF7E4AC0A80676380D413F43: done sorting span=0, length=20, time=1
2016-05-18 00:57:48,992 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DF9ED871BF7E4AC0A80676380D413F43: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:48,993 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DF9ED871BF7E4AC0A80676380D413F43: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001_wd/localmode-local-dir/output/attempt_1463533068766_0001_1_00_000000_0_10261_0/file.out
2016-05-18 00:57:48,998 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:48 Completed running task attempt: attempt_1463533068766_0001_1_00_000000_0
2016-05-18 00:57:48,999 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533068766_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:48,999 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533068766_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:48,999 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533068766_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533068989, Process_Duration=2, Process_End_Time=1463533068991, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:48,999 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533068766_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:49,000 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:49,000 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CB171CD582704EB9984809D454ED3C83, taskAttemptId=attempt_1463533068766_0001_1_00_000000_0, creationTime=1463533068937, allocationTime=1463533068939, startTime=1463533068940, finishTime=1463533069000, timeTaken=60, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:49,000 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:49,000 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,000 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533068766_0001_00_000001
2016-05-18 00:57:49,000 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533068766_0001_1_00_000000_0
2016-05-18 00:57:49,000 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:TASK_FINISHED]: vertexName=CB171CD582704EB9984809D454ED3C83, taskId=task_1463533068766_0001_1_00_000000, startTime=1463533068940, finishTime=1463533069000, timeTaken=60, status=SUCCEEDED, successfulAttemptID=attempt_1463533068766_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533068989, Process_Duration=2, Process_End_Time=1463533068991, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:49,001 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:49,001 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:49,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:VERTEX_FINISHED]: vertexName=CB171CD582704EB9984809D454ED3C83, vertexId=vertex_1463533068766_0001_1_00, initRequestedTime=1463533068922, initedTime=1463533068934, startRequestedTime=1463533068924, startedTime=1463533068934, finishTime=1463533069001, timeTaken=67, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533068989, Process_Duration=2, Process_End_Time=1463533068991, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533068940, firstTasksToStart=[ task_1463533068766_0001_1_00_000000 ], lastTaskFinishTime=1463533069000, lastTasksToFinish=[ task_1463533068766_0001_1_00_000000 ], minTaskDuration=60, maxTaskDuration=60, avgTaskDuration=60.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533068766_0001_1_00_000000 ], longestDurationTasks=[ task_1463533068766_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:49,002 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:49,002 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533068766_0001_00_000001, stoppedTime=1463533069002, exitStatus=0
2016-05-18 00:57:49,002 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43] attempt: attempt_1463533068766_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:49,002 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:49,002 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:49,002 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533068766_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:49,006 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DF9ED871BF7E4AC0A80676380D413F43
2016-05-18 00:57:49,006 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=CB171CD582704EB9984809D454ED3C83, destinationVertexName=DF9ED871BF7E4AC0A80676380D413F43. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:49,007 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]
2016-05-18 00:57:49,007 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:TASK_STARTED]: vertexName=DF9ED871BF7E4AC0A80676380D413F43, taskId=task_1463533068766_0001_1_01_000000, scheduledTime=1463533069007, launchTime=1463533069007
2016-05-18 00:57:49,007 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]
2016-05-18 00:57:49,009 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533068766_0001_00_000002, launchTime=1463533069009
2016-05-18 00:57:49,010 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533068766_0001_00_000002
2016-05-18 00:57:49,010 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533068766_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533068766_0001_1_01_000000_0
2016-05-18 00:57:49,010 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533068766_0001_1_01_000000_0] started. Is using containerId: [container_1463533068766_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:49,011 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533068766_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:49,011 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533068766_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:49,011 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533068766_0001_00_000001, Exiting
2016-05-18 00:57:49,011 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533068766_0001_00_000001
2016-05-18 00:57:49,011 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DF9ED871BF7E4AC0A80676380D413F43, taskAttemptId=attempt_1463533068766_0001_1_01_000000_0, startTime=1463533069010, containerId=container_1463533068766_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:49,011 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533068766_0001_00_000001
2016-05-18 00:57:49 Starting to run new task attempt: attempt_1463533068766_0001_1_01_000000_0
2016-05-18 00:57:49,011 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533068766_0001_1_01_000000_0
2016-05-18 00:57:49,011 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,012 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533068766_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:49,012 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533068766_0001_00_000001 completed successfully
2016-05-18 00:57:49,012 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:49,012 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC] every/(1/1) ...lyplatform/ALL_UNKNOWN_-1, VertexName: DF9ED871BF7E4AC0A80676380D413F43, VertexParallelism: 1, TaskAttemptID:attempt_1463533068766_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=CB171CD582704EB9984809D454ED3C83, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=AFD8CA50DE054A78997B3E905D26F519, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:49,012 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:49,013 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533068766_0001_1_01_000000_0
2016-05-18 00:57:49,013 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533068766_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]
2016-05-18 00:57:49,013 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533068766_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:49,016 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:49,019 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:49,019 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DF9ED871BF7E4AC0A80676380D413F43, ordinal: 1
2016-05-18 00:57:49,019 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DF9ED871BF7E4AC0A80676380D413F43, mem on start (mb), free: 170, total: 367, max: 672
2016-05-18 00:57:49,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:49,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:49,020 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:49,022 INFO output.MROutput (MROutput.java:initialize(421)) - AFD8CA50DE054A78997B3E905D26F519: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:49,023 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:49,023 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[AFD8CA50DE054A78997B3E905D26F519:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [CB171CD582704EB9984809D454ED3C83:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:49,023 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:49,023 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:49,023 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:49,024 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[AFD8CA50DE054A78997B3E905D26F519:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [CB171CD582704EB9984809D454ED3C83:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:49,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:49,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: CB171CD582704EB9984809D454ED3C83 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:49,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:49,024 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:49,024 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - CB171CD582704EB9984809D454ED3C83: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:49,024 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - CB171CD582704EB9984809D454ED3C83: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:49,025 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: CB171CD582704EB9984809D454ED3C83 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:49,026 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: CB171CD582704EB9984809D454ED3C83
2016-05-18 00:57:49,026 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:49,026 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533068766_0001_1_01_000000_0
2016-05-18 00:57:49,028 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CB171CD582704EB9984809D454ED3C83: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:49,028 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:49,028 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: B969D98C6BEB4BF388C5EF1E647E1F00
2016-05-18 00:57:49,029 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_-1"], id: AFD8CA50DE054A78997B3E905D26F519
2016-05-18 00:57:49,029 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_-1"]
2016-05-18 00:57:49,029 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] B969D98C6BEB4BF388C5EF1E647E1F00, for 1 inputs
2016-05-18 00:57:49,030 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533068766_0001_1_00_000000_0_10261} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533069030, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:49,030 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : CB171CD582704EB9984809D454ED3C83
2016-05-18 00:57:49,030 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:49,030 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: CB171CD582704EB9984809D454ED3C83, wasInterrupted=false
2016-05-18 00:57:49,031 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:49,031 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:49,031 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - CB171CD582704EB9984809D454ED3C83: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:49,032 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:49,032 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001_wd/localmode-local-dir/output/attempt_1463533068766_0001_1_00_000000_0_10261/file.out, len=181, isLocal=true
2016-05-18 00:57:49,032 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:49,032 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:49,032 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:49,033 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:49,033 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : CB171CD582704EB9984809D454ED3C83
2016-05-18 00:57:49,033 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - CB171CD582704EB9984809D454ED3C83: Shuffle Runner thread complete
2016-05-18 00:57:49,033 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DF9ED871BF7E4AC0A80676380D413F43, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:49,034 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:49,034 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - CB171CD582704EB9984809D454ED3C83: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:49,038 INFO output.MROutput (MROutput.java:close(526)) - AFD8CA50DE054A78997B3E905D26F519 closed
2016-05-18 00:57:49,038 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533068766_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:49,039 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533068766_0001_1_01_000000_0
2016-05-18 00:57:49,039 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DF9ED871BF7E4AC0A80676380D413F43, mem on close (mb), free: 167, total: 367, max: 672
2016-05-18 00:57:49,039 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: CB171CD582704EB9984809D454ED3C83
2016-05-18 00:57:49,039 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - CB171CD582704EB9984809D454ED3C83: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:49,039 INFO output.MROutput (MROutput.java:close(526)) - AFD8CA50DE054A78997B3E905D26F519 closed
2016-05-18 00:57:49,039 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:49 Completed running task attempt: attempt_1463533068766_0001_1_01_000000_0
2016-05-18 00:57:49,039 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533068766_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:49,039 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533068766_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:49,040 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533068766_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533069033, Process_Duration=6, Process_End_Time=1463533069039, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:49,040 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533068766_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:49,040 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DF9ED871BF7E4AC0A80676380D413F43, taskAttemptId=attempt_1463533068766_0001_1_01_000000_0, creationTime=1463533069007, allocationTime=1463533069008, startTime=1463533069010, finishTime=1463533069040, timeTaken=30, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:49,041 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533068766_0001_1_01_000000_0
2016-05-18 00:57:49,040 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:49,041 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:TASK_FINISHED]: vertexName=DF9ED871BF7E4AC0A80676380D413F43, taskId=task_1463533068766_0001_1_01_000000, startTime=1463533069010, finishTime=1463533069041, timeTaken=31, status=SUCCEEDED, successfulAttemptID=attempt_1463533068766_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533069033, Process_Duration=6, Process_End_Time=1463533069039, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:49,041 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:49,041 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43]
2016-05-18 00:57:49,041 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:49,041 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,042 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533068766_0001_00_000002
2016-05-18 00:57:49,042 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:VERTEX_FINISHED]: vertexName=DF9ED871BF7E4AC0A80676380D413F43, vertexId=vertex_1463533068766_0001_1_01, initRequestedTime=1463533068924, initedTime=1463533068929, startRequestedTime=1463533068934, startedTime=1463533068935, finishTime=1463533069041, timeTaken=106, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533069033, Process_Duration=6, Process_End_Time=1463533069039, Read_Duration=3, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533069010, firstTasksToStart=[ task_1463533068766_0001_1_01_000000 ], lastTaskFinishTime=1463533069040, lastTasksToFinish=[ task_1463533068766_0001_1_01_000000 ], minTaskDuration=30, maxTaskDuration=30, avgTaskDuration=30.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533068766_0001_1_01_000000 ], longestDurationTasks=[ task_1463533068766_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:49,042 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:49,042 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533068766_0001_00_000002, stoppedTime=1463533069042, exitStatus=0
2016-05-18 00:57:49,042 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:49,042 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:49,043 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533068766_0001_1_00 [CB171CD582704EB9984809D454ED3C83]
2016-05-18 00:57:49,043 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533068766_0001_1
2016-05-18 00:57:49,043 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533068766_0001_1
2016-05-18 00:57:49,043 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533068766_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:49,043 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: AFD8CA50DE054A78997B3E905D26F519 for vertex: vertex_1463533068766_0001_1_01 [DF9ED871BF7E4AC0A80676380D413F43], outputName: AFD8CA50DE054A78997B3E905D26F519
2016-05-18 00:57:49,043 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533068766_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:49,049 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:AFD8CA50DE054A78997B3E905D26F519 of vertex/vertexGroup:DF9ED871BF7E4AC0A80676380D413F43 isVertexGroupOutput:false
2016-05-18 00:57:49,049 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:49,050 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533068766_0001_1, queueSize=0
2016-05-18 00:57:49,050 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533068766_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533068766_0001_1, startTime=1463533068922, finishTime=1463533069049, timeTaken=127, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=220, AM_GC_TIME_MILLIS=10, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066138022, Process_Duration=8, Process_End_Time=2927066138030, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:49,050 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533068766_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:49,051 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533068766_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:49 Completed Dag: dag_1463533068766_0001_1
2016-05-18 00:57:49,051 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533068766_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:49,051 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533068766_0001_1_post
2016-05-18 00:57:49,051 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,051 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:49,051 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:49,051 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC] every/(1/1) ...lyplatform/ALL_UNKNOWN_-1, with id=dag_1463533068766_0001_1
2016-05-18 00:57:49,051 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC] every/(1/1) ...lyplatform/ALL_UNKNOWN_-1, with id=dag_1463533068766_0001_1
2016-05-18 00:57:49,052 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533068766_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:49,052 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533068766_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:49,052 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533068766_0001_00_000002, Exiting
2016-05-18 00:57:49,052 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533068766_0001_00_000002
2016-05-18 00:57:49,052 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533068766_0001_00_000002
2016-05-18 00:57:49,052 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533068766_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:49,052 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533068766_0001_00_000002 completed successfully
2016-05-18 00:57:49,061 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/ALL_UNKNOWN_-1, applicationId=application_1463533068766_0001
2016-05-18 00:57:49,062 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:49,062 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:49,062 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:49,062 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:49,062 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:49,063 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:49,063 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:49,063 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:49,063 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:49,063 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:49,064 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:49,064 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:49,064 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:49,065 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:49,067 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/298A529AB4AB41DAA6FFF60136F964A5/BD5DFFB292CA4B3AA587D5B5F2EEA0EC/.tez/application_1463533068766_0001
2016-05-18 00:57:49,068 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_-1/_temporary
2016-05-18 00:57:49,068 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.320
2016-05-18 00:57:49,137 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_UNKNOWN_-1
2016-05-18 00:57:49,158 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:49,158 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:49,184 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.004
2016-05-18 00:57:49,185 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.004
2016-05-18 00:57:49,185 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:49,185 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:49,185 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_UNKNOWN_'field'
2016-05-18 00:57:49,188 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:49,188 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:49,207 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:49,207 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:49,236 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:49,241 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 00:57:49,241 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:49,242 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:49,242 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:49,256 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:49,261 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_RESULTS
2016-05-18 00:57:49,306 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:49,307 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:49,307 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_RESULTS"]
2016-05-18 00:57:49,307 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:49,308 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:49,308 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:49,308 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...tform/ALL_UNKNOWN_RESULTS
2016-05-18 00:57:49,308 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:49,320 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:49,324 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:49,325 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:49,325 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:49,333 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001 doesn't exist and is created
2016-05-18 00:57:49,347 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:49,347 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:49,348 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001_wd
2016-05-18 00:57:49,357 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533069325_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:49,357 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:49,357 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:49,358 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:49,358 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:49,359 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:49,359 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:49,359 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:49,360 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:49,360 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:49,361 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:49,361 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:49,361 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:49,361 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:49,361 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:49,361 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:49,362 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:49,362 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:49,362 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:49,362 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:49,363 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:49,364 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:49,364 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:49,364 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001_wd/localmode-log-dir
2016-05-18 00:57:49,365 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001_wd/localmode-log-dir/history.txt.appattempt_1463533069325_0001_000000, maxErrors=10
2016-05-18 00:57:49,365 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:49,365 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533069325_0001_000000, appSubmitTime=1463533069356, launchTime=1463533069356
2016-05-18 00:57:49,369 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:49,370 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:59423
2016-05-18 00:57:49,370 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533069325
2016-05-18 00:57:49,371 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533069325_0001_000000, startTime=1463533069371
2016-05-18 00:57:49,371 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:49,447 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:49,447 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:49,448 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...tform/ALL_UNKNOWN_RESULTS, applicationId=application_1463533069325_0001, dagName=[616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A] every/(1/1) ...tform/ALL_UNKNOWN_RESULTS
2016-05-18 00:57:49,473 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A] every/(1/1) ...tform/ALL_UNKNOWN_RESULTS
2016-05-18 00:57:49,473 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:49,474 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533069325_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001_wd/localmode-log-dir/dag_1463533069325_0001_1.dot
2016-05-18 00:57:49,474 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001_wd/localmode-log-dir/dag_1463533069325_0001_1-tez-dag.pb.txt
2016-05-18 00:57:49,480 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533069325_0001_1
2016-05-18 00:57:49,480 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,480 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A] every/(1/1) ...tform/ALL_UNKNOWN_RESULTS
2016-05-18 00:57:49 Running Dag: dag_1463533069325_0001_1
2016-05-18 00:57:49,489 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533069325_0001_1, submitTime=1463533069473
2016-05-18 00:57:49,490 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001/tez-conf.pb" } size: 36913 timestamp: 1463533069000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:49,490 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,490 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:49,491 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001/tez-conf.pb" } size: 36913 timestamp: 1463533069000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:49,491 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]
2016-05-18 00:57:49,491 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:49,491 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:49,491 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533069325_0001_1, initTime=1463533069491
2016-05-18 00:57:49,491 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533069325_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:49,491 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:49,492 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:DAG_STARTED]: dagID=dag_1463533069325_0001_1, startTime=1463533069492
2016-05-18 00:57:49,492 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...tform/ALL_UNKNOWN_RESULTS, applicationId=application_1463533069325_0001, dagName=[616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A] every/(1/1) ...tform/ALL_UNKNOWN_RESULTS
2016-05-18 00:57:49,492 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533069325_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:49,492 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: B8D4D417DA86470E9317C86F5AF1B0A8 : {29B0734B3FC34D07A5FB09793E102CD5={InputName=29B0734B3FC34D07A5FB09793E102CD5}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:49,492 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 29B0734B3FC34D07A5FB09793E102CD5, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:49,492 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,492 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,492 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,493 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,493 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:49,493 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]
2016-05-18 00:57:49,495 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 29B0734B3FC34D07A5FB09793E102CD5 on vertex vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,496 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:49,496 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]
2016-05-18 00:57:49,496 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]
2016-05-18 00:57:49,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533069325_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:49,497 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F], numAdditionalOutputs=1
2016-05-18 00:57:49,497 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=5BCDDFEE2AC0490B8814D4C52F326892, vertex=vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:49,498 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 29B0734B3FC34D07A5FB09793E102CD5 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:49,499 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for B768753B272E4537BE9C238D5B20F93F:5BCDDFEE2AC0490B8814D4C52F326892 using oldmapred API
2016-05-18 00:57:49,499 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=5BCDDFEE2AC0490B8814D4C52F326892, vertexName=B768753B272E4537BE9C238D5B20F93F, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:49,500 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:49,500 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:49,500 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B768753B272E4537BE9C238D5B20F93F, vertexId=vertex_1463533069325_0001_1_01, initRequestedTime=1463533069493, initedTime=1463533069497, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:49,500 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:49,501 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:49,501 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:49,501 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 29B0734B3FC34D07A5FB09793E102CD5 on vertex vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,502 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:49,502 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {29B0734B3FC34D07A5FB09793E102CD5=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:49,502 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] parallelism set to 1
2016-05-18 00:57:49,502 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:49,503 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B8D4D417DA86470E9317C86F5AF1B0A8, vertexId=vertex_1463533069325_0001_1_00, initRequestedTime=1463533069492, initedTime=1463533069503, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:49,503 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] with distanceFromRoot: 0
2016-05-18 00:57:49,503 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:49,503 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533069325_0001_1_00, startRequestedTime=1463533069493, startedTime=1463533069503
2016-05-18 00:57:49,503 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533069325_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:49,503 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:49,503 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533069325_0001_1_00 for vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F] numStartedSources: 1 numSources: 1
2016-05-18 00:57:49,503 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F] with distanceFromRoot: 1
2016-05-18 00:57:49,504 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533069325_0001_1_01, startRequestedTime=1463533069503, startedTime=1463533069504
2016-05-18 00:57:49,504 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:49,505 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: B768753B272E4537BE9C238D5B20F93F with 0 source tasks and 1 pending tasks
2016-05-18 00:57:49,505 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: B8D4D417DA86470E9317C86F5AF1B0A8 in vertex: B768753B272E4537BE9C238D5B20F93F numBipartiteSourceTasks: 1
2016-05-18 00:57:49,505 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in B8D4D417DA86470E9317C86F5AF1B0A8
2016-05-18 00:57:49,506 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,506 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:TASK_STARTED]: vertexName=B8D4D417DA86470E9317C86F5AF1B0A8, taskId=task_1463533069325_0001_1_00_000000, scheduledTime=1463533069506, launchTime=1463533069506
2016-05-18 00:57:49,506 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,507 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:49,508 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533069325_0001_00_000001, launchTime=1463533069508
2016-05-18 00:57:49,509 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533069325_0001_00_000001
2016-05-18 00:57:49,509 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533069325_0001_1_00_000000_0] started. Is using containerId: [container_1463533069325_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:49,509 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B8D4D417DA86470E9317C86F5AF1B0A8, taskAttemptId=attempt_1463533069325_0001_1_00_000000_0, startTime=1463533069509, containerId=container_1463533069325_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:49,510 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533069325_0001_00_000001: 1 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533069325_0001_1_00_000000_0
2016-05-18 00:57:49 Starting to run new task attempt: attempt_1463533069325_0001_1_00_000000_0
2016-05-18 00:57:49,510 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533069325_0001_1_00_000000_0
2016-05-18 00:57:49,510 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,510 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:49,510 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A] every/(1/1) ...tform/ALL_UNKNOWN_RESULTS, VertexName: B8D4D417DA86470E9317C86F5AF1B0A8, VertexParallelism: 1, TaskAttemptID:attempt_1463533069325_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=29B0734B3FC34D07A5FB09793E102CD5, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B768753B272E4537BE9C238D5B20F93F, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:49,510 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:49,511 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533069325_0001_1_00_000000_0
2016-05-18 00:57:49,512 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533069325_0001, with dag id: dag_1463533069325_0001_1
2016-05-18 00:57:49,512 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533069325_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:49,513 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533069325_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:49,515 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:49,518 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B8D4D417DA86470E9317C86F5AF1B0A8, ordinal: 0
2016-05-18 00:57:49,518 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8D4D417DA86470E9317C86F5AF1B0A8, mem on start (mb), free: 257, total: 366, max: 672
2016-05-18 00:57:49,518 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:49,518 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:49,518 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:49,518 INFO input.MRInput (MRInput.java:initialize(440)) - 29B0734B3FC34D07A5FB09793E102CD5 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:49,518 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 29B0734B3FC34D07A5FB09793E102CD5
2016-05-18 00:57:49,519 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:49,519 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[29B0734B3FC34D07A5FB09793E102CD5:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [B768753B272E4537BE9C238D5B20F93F:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:49,519 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:49,519 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:49,519 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:49,519 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[29B0734B3FC34D07A5FB09793E102CD5:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [B768753B272E4537BE9C238D5B20F93F:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:49,519 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:49,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 29B0734B3FC34D07A5FB09793E102CD5 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:49,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:49,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:49,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 29B0734B3FC34D07A5FB09793E102CD5
2016-05-18 00:57:49,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:49,520 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533069325_0001_1_00_000000_0
2016-05-18 00:57:49,521 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:49,522 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 29B0734B3FC34D07A5FB09793E102CD5 initialized RecordReader from event
2016-05-18 00:57:49,523 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:49,523 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 29B0734B3FC34D07A5FB09793E102CD5
2016-05-18 00:57:49,523 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: EEF3C274628B4F4AA975714681162D24
2016-05-18 00:57:49,523 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] EEF3C274628B4F4AA975714681162D24
2016-05-18 00:57:49,524 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:49,526 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:49,526 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - B768753B272E4537BE9C238D5B20F93F using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@2e80ad33, comparator=cascading.tuple.hadoop.util.TupleComparator@1eb071f, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:49,526 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for B768753B272E4537BE9C238D5B20F93F: , UsingHashComparator=false
2016-05-18 00:57:49,539 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:49,539 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:49,539 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for B768753B272E4537BE9C238D5B20F93F: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:49,539 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - B768753B272E4537BE9C238D5B20F93F: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:49,539 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:49,540 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B8D4D417DA86470E9317C86F5AF1B0A8, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:49,542 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533069325_0001_1_00_000000_0
2016-05-18 00:57:49,542 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B8D4D417DA86470E9317C86F5AF1B0A8, mem on close (mb), free: 154, total: 366, max: 672
2016-05-18 00:57:49,542 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - B768753B272E4537BE9C238D5B20F93F: Starting flush of map output
2016-05-18 00:57:49,543 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - B768753B272E4537BE9C238D5B20F93F: Span0.length = 20, perItem = 6
2016-05-18 00:57:49,543 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - B768753B272E4537BE9C238D5B20F93F: done sorting span=0, length=20, time=0
2016-05-18 00:57:49,543 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - B768753B272E4537BE9C238D5B20F93F: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:49,544 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - B768753B272E4537BE9C238D5B20F93F: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001_wd/localmode-local-dir/output/attempt_1463533069325_0001_1_00_000000_0_10267_0/file.out
2016-05-18 00:57:49,547 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:49 Completed running task attempt: attempt_1463533069325_0001_1_00_000000_0
2016-05-18 00:57:49,547 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533069325_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:49,548 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533069325_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:49,548 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533069325_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533069540, Process_Duration=2, Process_End_Time=1463533069542, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:49,548 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533069325_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:49,549 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:49,549 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B8D4D417DA86470E9317C86F5AF1B0A8, taskAttemptId=attempt_1463533069325_0001_1_00_000000_0, creationTime=1463533069506, allocationTime=1463533069508, startTime=1463533069509, finishTime=1463533069549, timeTaken=40, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:49,549 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533069325_0001_1_00_000000_0
2016-05-18 00:57:49,549 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:49,549 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,549 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:TASK_FINISHED]: vertexName=B8D4D417DA86470E9317C86F5AF1B0A8, taskId=task_1463533069325_0001_1_00_000000, startTime=1463533069509, finishTime=1463533069549, timeTaken=40, status=SUCCEEDED, successfulAttemptID=attempt_1463533069325_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533069540, Process_Duration=2, Process_End_Time=1463533069542, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:49,550 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533069325_0001_00_000001
2016-05-18 00:57:49,550 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:49,550 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,551 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:VERTEX_FINISHED]: vertexName=B8D4D417DA86470E9317C86F5AF1B0A8, vertexId=vertex_1463533069325_0001_1_00, initRequestedTime=1463533069492, initedTime=1463533069503, startRequestedTime=1463533069493, startedTime=1463533069503, finishTime=1463533069550, timeTaken=47, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533069540, Process_Duration=2, Process_End_Time=1463533069542, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533069509, firstTasksToStart=[ task_1463533069325_0001_1_00_000000 ], lastTaskFinishTime=1463533069549, lastTasksToFinish=[ task_1463533069325_0001_1_00_000000 ], minTaskDuration=40, maxTaskDuration=40, avgTaskDuration=40.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533069325_0001_1_00_000000 ], longestDurationTasks=[ task_1463533069325_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:49,551 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:49,551 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533069325_0001_00_000001, stoppedTime=1463533069551, exitStatus=0
2016-05-18 00:57:49,551 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F] attempt: attempt_1463533069325_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:49,552 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:49,552 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:49,552 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B768753B272E4537BE9C238D5B20F93F
2016-05-18 00:57:49,552 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533069325_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:49,552 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=B8D4D417DA86470E9317C86F5AF1B0A8, destinationVertexName=B768753B272E4537BE9C238D5B20F93F. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:49,552 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]
2016-05-18 00:57:49,552 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:TASK_STARTED]: vertexName=B768753B272E4537BE9C238D5B20F93F, taskId=task_1463533069325_0001_1_01_000000, scheduledTime=1463533069552, launchTime=1463533069552
2016-05-18 00:57:49,553 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]
2016-05-18 00:57:49,554 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533069325_0001_00_000002, launchTime=1463533069554
2016-05-18 00:57:49,555 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533069325_0001_00_000002
2016-05-18 00:57:49,555 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533069325_0001_1_01_000000_0] started. Is using containerId: [container_1463533069325_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:49,555 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B768753B272E4537BE9C238D5B20F93F, taskAttemptId=attempt_1463533069325_0001_1_01_000000_0, startTime=1463533069555, containerId=container_1463533069325_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:49,556 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533069325_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533069325_0001_1_01_000000_0
2016-05-18 00:57:49 Starting to run new task attempt: attempt_1463533069325_0001_1_01_000000_0
2016-05-18 00:57:49,556 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533069325_0001_1_01_000000_0
2016-05-18 00:57:49,556 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,556 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:49,556 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A] every/(1/1) ...tform/ALL_UNKNOWN_RESULTS, VertexName: B768753B272E4537BE9C238D5B20F93F, VertexParallelism: 1, TaskAttemptID:attempt_1463533069325_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B8D4D417DA86470E9317C86F5AF1B0A8, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=5BCDDFEE2AC0490B8814D4C52F326892, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:49,556 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:49,557 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533069325_0001_1_01_000000_0
2016-05-18 00:57:49,558 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533069325_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]
2016-05-18 00:57:49,559 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533069325_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:49,559 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:49,560 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:49,560 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533069325_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:49,560 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533069325_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:49,560 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533069325_0001_00_000001, Exiting
2016-05-18 00:57:49,560 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533069325_0001_00_000001
2016-05-18 00:57:49,561 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533069325_0001_00_000001
2016-05-18 00:57:49,564 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533069325_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:49,564 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533069325_0001_00_000001 completed successfully
2016-05-18 00:57:49,565 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B768753B272E4537BE9C238D5B20F93F, ordinal: 1
2016-05-18 00:57:49,565 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B768753B272E4537BE9C238D5B20F93F, mem on start (mb), free: 150, total: 366, max: 672
2016-05-18 00:57:49,565 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:49,565 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:49,565 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:49,569 INFO output.MROutput (MROutput.java:initialize(421)) - 5BCDDFEE2AC0490B8814D4C52F326892: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:49,569 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:49,569 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B8D4D417DA86470E9317C86F5AF1B0A8:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [5BCDDFEE2AC0490B8814D4C52F326892:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:49,569 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:49,569 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:49,570 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:49,570 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B8D4D417DA86470E9317C86F5AF1B0A8:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [5BCDDFEE2AC0490B8814D4C52F326892:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:49,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:49,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B8D4D417DA86470E9317C86F5AF1B0A8 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:49,570 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - B8D4D417DA86470E9317C86F5AF1B0A8: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:49,570 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - B8D4D417DA86470E9317C86F5AF1B0A8: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:49,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:49,571 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:49,571 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: B8D4D417DA86470E9317C86F5AF1B0A8 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:49,572 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B8D4D417DA86470E9317C86F5AF1B0A8
2016-05-18 00:57:49,572 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:49,572 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533069325_0001_1_01_000000_0
2016-05-18 00:57:49,574 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8D4D417DA86470E9317C86F5AF1B0A8: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:49,574 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:49,575 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: EEF3C274628B4F4AA975714681162D24
2016-05-18 00:57:49,575 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_RESULTS"], id: 5BCDDFEE2AC0490B8814D4C52F326892
2016-05-18 00:57:49,575 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_RESULTS"]
2016-05-18 00:57:49,575 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] EEF3C274628B4F4AA975714681162D24, for 1 inputs
2016-05-18 00:57:49,576 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533069325_0001_1_00_000000_0_10267} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533069576, TimeTaken=2, Rate=0.09 MB/s
2016-05-18 00:57:49,576 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : B8D4D417DA86470E9317C86F5AF1B0A8
2016-05-18 00:57:49,576 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:49,577 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: B8D4D417DA86470E9317C86F5AF1B0A8, wasInterrupted=false
2016-05-18 00:57:49,577 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:49,579 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:49,580 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:49,580 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001_wd/localmode-local-dir/output/attempt_1463533069325_0001_1_00_000000_0_10267/file.out, len=181, isLocal=true
2016-05-18 00:57:49,580 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:49,580 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:49,580 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:49,581 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:49,582 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : B8D4D417DA86470E9317C86F5AF1B0A8
2016-05-18 00:57:49,582 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - B8D4D417DA86470E9317C86F5AF1B0A8: Shuffle Runner thread complete
2016-05-18 00:57:49,582 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B768753B272E4537BE9C238D5B20F93F, all 1 inputs ready in: 00:00:00.007
2016-05-18 00:57:49,583 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:49,583 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - B8D4D417DA86470E9317C86F5AF1B0A8: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:49,587 INFO output.MROutput (MROutput.java:close(526)) - 5BCDDFEE2AC0490B8814D4C52F326892 closed
2016-05-18 00:57:49,587 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533069325_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:49,588 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533069325_0001_1_01_000000_0
2016-05-18 00:57:49,588 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B768753B272E4537BE9C238D5B20F93F, mem on close (mb), free: 147, total: 366, max: 672
2016-05-18 00:57:49,588 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: B8D4D417DA86470E9317C86F5AF1B0A8
2016-05-18 00:57:49,588 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - B8D4D417DA86470E9317C86F5AF1B0A8: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:49,588 INFO output.MROutput (MROutput.java:close(526)) - 5BCDDFEE2AC0490B8814D4C52F326892 closed
2016-05-18 00:57:49,589 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:49 Completed running task attempt: attempt_1463533069325_0001_1_01_000000_0
2016-05-18 00:57:49,589 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533069325_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:49,590 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533069325_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:49,590 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533069325_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533069582, Process_Duration=6, Process_End_Time=1463533069588, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:49,591 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533069325_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:49,591 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:49,591 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B768753B272E4537BE9C238D5B20F93F, taskAttemptId=attempt_1463533069325_0001_1_01_000000_0, creationTime=1463533069553, allocationTime=1463533069553, startTime=1463533069555, finishTime=1463533069591, timeTaken=36, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:49,591 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:49,591 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,591 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533069325_0001_1_01_000000_0
2016-05-18 00:57:49,591 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533069325_0001_00_000002
2016-05-18 00:57:49,592 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:TASK_FINISHED]: vertexName=B768753B272E4537BE9C238D5B20F93F, taskId=task_1463533069325_0001_1_01_000000, startTime=1463533069555, finishTime=1463533069591, timeTaken=36, status=SUCCEEDED, successfulAttemptID=attempt_1463533069325_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533069582, Process_Duration=6, Process_End_Time=1463533069588, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:49,592 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:49,592 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F]
2016-05-18 00:57:49,593 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:VERTEX_FINISHED]: vertexName=B768753B272E4537BE9C238D5B20F93F, vertexId=vertex_1463533069325_0001_1_01, initRequestedTime=1463533069493, initedTime=1463533069497, startRequestedTime=1463533069503, startedTime=1463533069504, finishTime=1463533069592, timeTaken=88, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533069582, Process_Duration=6, Process_End_Time=1463533069588, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533069555, firstTasksToStart=[ task_1463533069325_0001_1_01_000000 ], lastTaskFinishTime=1463533069591, lastTasksToFinish=[ task_1463533069325_0001_1_01_000000 ], minTaskDuration=36, maxTaskDuration=36, avgTaskDuration=36.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533069325_0001_1_01_000000 ], longestDurationTasks=[ task_1463533069325_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:49,593 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:49,593 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533069325_0001_00_000002, stoppedTime=1463533069593, exitStatus=0
2016-05-18 00:57:49,593 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:49,593 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:49,593 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533069325_0001_1_00 [B8D4D417DA86470E9317C86F5AF1B0A8]
2016-05-18 00:57:49,593 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533069325_0001_1
2016-05-18 00:57:49,594 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533069325_0001_1
2016-05-18 00:57:49,594 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533069325_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:49,594 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: 5BCDDFEE2AC0490B8814D4C52F326892 for vertex: vertex_1463533069325_0001_1_01 [B768753B272E4537BE9C238D5B20F93F], outputName: 5BCDDFEE2AC0490B8814D4C52F326892
2016-05-18 00:57:49,594 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533069325_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:49,599 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:5BCDDFEE2AC0490B8814D4C52F326892 of vertex/vertexGroup:B768753B272E4537BE9C238D5B20F93F isVertexGroupOutput:false
2016-05-18 00:57:49,599 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:49,600 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533069325_0001_1, queueSize=0
2016-05-18 00:57:49,601 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069325_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533069325_0001_1, startTime=1463533069491, finishTime=1463533069599, timeTaken=108, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=170, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066139122, Process_Duration=8, Process_End_Time=2927066139130, Read_Duration=1, Tuples_Read=40, Tuples_Written=40, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:49,601 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533069325_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:49,601 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533069325_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:49 Completed Dag: dag_1463533069325_0001_1
2016-05-18 00:57:49,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533069325_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:49,601 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533069325_0001_1_post
2016-05-18 00:57:49,601 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:49,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:49,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A] every/(1/1) ...tform/ALL_UNKNOWN_RESULTS, with id=dag_1463533069325_0001_1
2016-05-18 00:57:49,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A] every/(1/1) ...tform/ALL_UNKNOWN_RESULTS, with id=dag_1463533069325_0001_1
2016-05-18 00:57:49,602 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533069325_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:49,602 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533069325_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:49,602 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533069325_0001_00_000002, Exiting
2016-05-18 00:57:49,602 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533069325_0001_00_000002
2016-05-18 00:57:49,602 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533069325_0001_00_000002
2016-05-18 00:57:49,602 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533069325_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:49,603 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533069325_0001_00_000002 completed successfully
2016-05-18 00:57:49,608 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...tform/ALL_UNKNOWN_RESULTS, applicationId=application_1463533069325_0001
2016-05-18 00:57:49,609 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:49,609 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:49,610 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:49,610 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:49,610 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:49,610 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:49,610 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:49,610 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:49,611 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:49,611 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:49,611 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:49,611 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:49,611 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:49,612 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:49,614 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/616A66AF4A96488EA5C138691DA3F990/A9AA9A52B682443593D62369AD6EEA6A/.tez/application_1463533069325_0001
2016-05-18 00:57:49,615 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_RESULTS/_temporary
2016-05-18 00:57:49,615 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.308
2016-05-18 00:57:49,681 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_UNKNOWN_RESULTS
2016-05-18 00:57:49,702 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:49,703 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:49,732 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:49,736 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.010
2016-05-18 00:57:49,736 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:49,736 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:49,737 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:49,750 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:49,755 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_ALL
2016-05-18 00:57:49,765 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:49,765 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:49,798 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:49,798 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:49,799 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_ALL"]
2016-05-18 00:57:49,799 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:49,799 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:49,799 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:49,799 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...yplatform/ALL_UNKNOWN_ALL
2016-05-18 00:57:49,799 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:49,813 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:49,817 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:49,817 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:49,818 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:49,826 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001 doesn't exist and is created
2016-05-18 00:57:49,840 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:49,841 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:49,841 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001_wd
2016-05-18 00:57:49,850 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533069817_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:49,851 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:49,851 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:49,852 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:49,852 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:49,852 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:49,852 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:49,853 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:49,854 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:49,854 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:49,854 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:49,855 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:49,855 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:49,855 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:49,855 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:49,855 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:49,855 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:49,855 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:49,855 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:49,855 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:49,857 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:49,858 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:49,858 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:49,858 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001_wd/localmode-log-dir
2016-05-18 00:57:49,858 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001_wd/localmode-log-dir/history.txt.appattempt_1463533069817_0001_000000, maxErrors=10
2016-05-18 00:57:49,859 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:49,859 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533069817_0001_000000, appSubmitTime=1463533069850, launchTime=1463533069850
2016-05-18 00:57:49,862 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:49,864 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:56571
2016-05-18 00:57:49,864 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533069817
2016-05-18 00:57:49,866 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533069817_0001_000000, startTime=1463533069866
2016-05-18 00:57:49,866 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:49,941 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:49,941 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:49,941 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...yplatform/ALL_UNKNOWN_ALL, applicationId=application_1463533069817_0001, dagName=[8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39] every/(1/1) ...yplatform/ALL_UNKNOWN_ALL
2016-05-18 00:57:49,966 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39] every/(1/1) ...yplatform/ALL_UNKNOWN_ALL
2016-05-18 00:57:49,966 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:49,967 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533069817_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001_wd/localmode-log-dir/dag_1463533069817_0001_1.dot
2016-05-18 00:57:49,967 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001_wd/localmode-log-dir/dag_1463533069817_0001_1-tez-dag.pb.txt
2016-05-18 00:57:49,976 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533069817_0001_1
2016-05-18 00:57:49,976 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:49,976 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39] every/(1/1) ...yplatform/ALL_UNKNOWN_ALL
2016-05-18 00:57:49 Running Dag: dag_1463533069817_0001_1
2016-05-18 00:57:49,986 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533069817_0001_1, submitTime=1463533069966
2016-05-18 00:57:49,987 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001/tez-conf.pb" } size: 36913 timestamp: 1463533069000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:49,987 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:49,987 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:49,989 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001/tez-conf.pb" } size: 36913 timestamp: 1463533069000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:49,989 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]
2016-05-18 00:57:49,989 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:49,989 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:49,990 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533069817_0001_1, initTime=1463533069990
2016-05-18 00:57:49,991 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533069817_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:49,991 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:49,991 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:DAG_STARTED]: dagID=dag_1463533069817_0001_1, startTime=1463533069991
2016-05-18 00:57:49,991 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533069817_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:49,991 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...yplatform/ALL_UNKNOWN_ALL, applicationId=application_1463533069817_0001, dagName=[8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39] every/(1/1) ...yplatform/ALL_UNKNOWN_ALL
2016-05-18 00:57:49,991 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: BC236D77EB5F41CDB7FFEEFB70D82DCF : {B466B4D188AE41ADB9448FF33E83AAFB={InputName=B466B4D188AE41ADB9448FF33E83AAFB}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:49,991 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: B466B4D188AE41ADB9448FF33E83AAFB, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:49,991 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:49,992 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:49,992 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:49,992 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:49,992 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:49,992 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]
2016-05-18 00:57:49,994 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: B466B4D188AE41ADB9448FF33E83AAFB on vertex vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:49,995 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:49,995 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]
2016-05-18 00:57:49,995 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]
2016-05-18 00:57:49,995 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533069817_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:49,996 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94], numAdditionalOutputs=1
2016-05-18 00:57:49,996 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=C2979CB7812E467882E55FFC63C02544, vertex=vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:49,996 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input B466B4D188AE41ADB9448FF33E83AAFB asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:49,997 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:49,998 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:49,998 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:49,998 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:49,998 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for B21BEE7D04E44D3698724AE23BB1CA94:C2979CB7812E467882E55FFC63C02544 using oldmapred API
2016-05-18 00:57:49,998 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=C2979CB7812E467882E55FFC63C02544, vertexName=B21BEE7D04E44D3698724AE23BB1CA94, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:49,998 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: B466B4D188AE41ADB9448FF33E83AAFB on vertex vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:49,999 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B21BEE7D04E44D3698724AE23BB1CA94, vertexId=vertex_1463533069817_0001_1_01, initRequestedTime=1463533069992, initedTime=1463533069996, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:49,999 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:50,000 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:50,000 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {B466B4D188AE41ADB9448FF33E83AAFB=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:50,000 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] parallelism set to 1
2016-05-18 00:57:50,000 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:50,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BC236D77EB5F41CDB7FFEEFB70D82DCF, vertexId=vertex_1463533069817_0001_1_00, initRequestedTime=1463533069991, initedTime=1463533070001, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:50,001 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] with distanceFromRoot: 0
2016-05-18 00:57:50,001 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:50,001 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533069817_0001_1_00, startRequestedTime=1463533069992, startedTime=1463533070001
2016-05-18 00:57:50,002 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533069817_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:50,002 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:50,002 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533069817_0001_1_00 for vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94] numStartedSources: 1 numSources: 1
2016-05-18 00:57:50,002 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94] with distanceFromRoot: 1
2016-05-18 00:57:50,002 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in BC236D77EB5F41CDB7FFEEFB70D82DCF
2016-05-18 00:57:50,002 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:50,003 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533069817_0001_1_01, startRequestedTime=1463533070002, startedTime=1463533070002
2016-05-18 00:57:50,003 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:50,003 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:TASK_STARTED]: vertexName=BC236D77EB5F41CDB7FFEEFB70D82DCF, taskId=task_1463533069817_0001_1_00_000000, scheduledTime=1463533070003, launchTime=1463533070003
2016-05-18 00:57:50,004 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:50,004 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: B21BEE7D04E44D3698724AE23BB1CA94 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:50,004 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BC236D77EB5F41CDB7FFEEFB70D82DCF in vertex: B21BEE7D04E44D3698724AE23BB1CA94 numBipartiteSourceTasks: 1
2016-05-18 00:57:50,004 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:50,006 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533069817_0001_00_000001, launchTime=1463533070006
2016-05-18 00:57:50,007 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533069817_0001_00_000001
2016-05-18 00:57:50,007 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533069817_0001_1_00_000000_0] started. Is using containerId: [container_1463533069817_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:50,007 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BC236D77EB5F41CDB7FFEEFB70D82DCF, taskAttemptId=attempt_1463533069817_0001_1_00_000000_0, startTime=1463533070007, containerId=container_1463533069817_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:50,008 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533069817_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533069817_0001_1_00_000000_0
2016-05-18 00:57:50 Starting to run new task attempt: attempt_1463533069817_0001_1_00_000000_0
2016-05-18 00:57:50,009 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533069817_0001_1_00_000000_0
2016-05-18 00:57:50,009 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,009 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:50,010 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39] every/(1/1) ...yplatform/ALL_UNKNOWN_ALL, VertexName: BC236D77EB5F41CDB7FFEEFB70D82DCF, VertexParallelism: 1, TaskAttemptID:attempt_1463533069817_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B466B4D188AE41ADB9448FF33E83AAFB, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=B21BEE7D04E44D3698724AE23BB1CA94, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:50,010 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533069817_0001, with dag id: dag_1463533069817_0001_1
2016-05-18 00:57:50,011 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:50,011 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533069817_0001_1_00_000000_0
2016-05-18 00:57:50,014 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:50,014 INFO input.MRInput (MRInput.java:initialize(440)) - B466B4D188AE41ADB9448FF33E83AAFB using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:50,014 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: B466B4D188AE41ADB9448FF33E83AAFB
2016-05-18 00:57:50,014 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533069817_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:50,015 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533069817_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:50,019 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BC236D77EB5F41CDB7FFEEFB70D82DCF, ordinal: 0
2016-05-18 00:57:50,019 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BC236D77EB5F41CDB7FFEEFB70D82DCF, mem on start (mb), free: 255, total: 366, max: 672
2016-05-18 00:57:50,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:50,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:50,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:50,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:50,019 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[B466B4D188AE41ADB9448FF33E83AAFB:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [B21BEE7D04E44D3698724AE23BB1CA94:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:50,019 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:50,019 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:50,019 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:50,020 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[B466B4D188AE41ADB9448FF33E83AAFB:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [B21BEE7D04E44D3698724AE23BB1CA94:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:50,020 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:50,020 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B466B4D188AE41ADB9448FF33E83AAFB being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:50,020 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B466B4D188AE41ADB9448FF33E83AAFB
2016-05-18 00:57:50,020 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:50,020 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:50,020 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:50,021 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533069817_0001_1_00_000000_0
2016-05-18 00:57:50,022 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:50,022 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - B466B4D188AE41ADB9448FF33E83AAFB initialized RecordReader from event
2016-05-18 00:57:50,023 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:50,024 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: B466B4D188AE41ADB9448FF33E83AAFB
2016-05-18 00:57:50,024 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 6B63F16A14284C92B38085996592E145
2016-05-18 00:57:50,024 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 6B63F16A14284C92B38085996592E145
2016-05-18 00:57:50,025 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,027 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,027 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - B21BEE7D04E44D3698724AE23BB1CA94 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@442041a8, comparator=cascading.tuple.hadoop.util.TupleComparator@21b69f89, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:50,027 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for B21BEE7D04E44D3698724AE23BB1CA94: , UsingHashComparator=false
2016-05-18 00:57:50,040 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:50,040 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:50,040 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for B21BEE7D04E44D3698724AE23BB1CA94: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:50,040 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - B21BEE7D04E44D3698724AE23BB1CA94: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:50,041 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:50,041 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BC236D77EB5F41CDB7FFEEFB70D82DCF, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:50,043 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533069817_0001_1_00_000000_0
2016-05-18 00:57:50,043 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BC236D77EB5F41CDB7FFEEFB70D82DCF, mem on close (mb), free: 153, total: 366, max: 672
2016-05-18 00:57:50,043 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - B21BEE7D04E44D3698724AE23BB1CA94: Starting flush of map output
2016-05-18 00:57:50,043 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - B21BEE7D04E44D3698724AE23BB1CA94: Span0.length = 20, perItem = 6
2016-05-18 00:57:50,043 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - B21BEE7D04E44D3698724AE23BB1CA94: done sorting span=0, length=20, time=0
2016-05-18 00:57:50,044 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - B21BEE7D04E44D3698724AE23BB1CA94: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:50,045 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - B21BEE7D04E44D3698724AE23BB1CA94: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001_wd/localmode-local-dir/output/attempt_1463533069817_0001_1_00_000000_0_10273_0/file.out
2016-05-18 00:57:50,047 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:50 Completed running task attempt: attempt_1463533069817_0001_1_00_000000_0
2016-05-18 00:57:50,048 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533069817_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:50,048 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533069817_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:50,048 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533069817_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533070041, Process_Duration=2, Process_End_Time=1463533070043, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:50,049 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533069817_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:50,049 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:50,049 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BC236D77EB5F41CDB7FFEEFB70D82DCF, taskAttemptId=attempt_1463533069817_0001_1_00_000000_0, creationTime=1463533070003, allocationTime=1463533070005, startTime=1463533070007, finishTime=1463533070049, timeTaken=42, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:50,049 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:50,049 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,049 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533069817_0001_00_000001
2016-05-18 00:57:50,049 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533069817_0001_1_00_000000_0
2016-05-18 00:57:50,050 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:TASK_FINISHED]: vertexName=BC236D77EB5F41CDB7FFEEFB70D82DCF, taskId=task_1463533069817_0001_1_00_000000, startTime=1463533070007, finishTime=1463533070049, timeTaken=42, status=SUCCEEDED, successfulAttemptID=attempt_1463533069817_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533070041, Process_Duration=2, Process_End_Time=1463533070043, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:50,050 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:50,050 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:50,051 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:VERTEX_FINISHED]: vertexName=BC236D77EB5F41CDB7FFEEFB70D82DCF, vertexId=vertex_1463533069817_0001_1_00, initRequestedTime=1463533069991, initedTime=1463533070001, startRequestedTime=1463533069992, startedTime=1463533070001, finishTime=1463533070050, timeTaken=49, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533070041, Process_Duration=2, Process_End_Time=1463533070043, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533070007, firstTasksToStart=[ task_1463533069817_0001_1_00_000000 ], lastTaskFinishTime=1463533070049, lastTasksToFinish=[ task_1463533069817_0001_1_00_000000 ], minTaskDuration=42, maxTaskDuration=42, avgTaskDuration=42.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533069817_0001_1_00_000000 ], longestDurationTasks=[ task_1463533069817_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:50,051 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:50,051 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533069817_0001_00_000001, stoppedTime=1463533070051, exitStatus=0
2016-05-18 00:57:50,051 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94] attempt: attempt_1463533069817_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:50,051 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:50,051 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:50,051 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533069817_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:50,052 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: B21BEE7D04E44D3698724AE23BB1CA94
2016-05-18 00:57:50,052 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BC236D77EB5F41CDB7FFEEFB70D82DCF, destinationVertexName=B21BEE7D04E44D3698724AE23BB1CA94. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:50,052 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]
2016-05-18 00:57:50,052 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:TASK_STARTED]: vertexName=B21BEE7D04E44D3698724AE23BB1CA94, taskId=task_1463533069817_0001_1_01_000000, scheduledTime=1463533070052, launchTime=1463533070052
2016-05-18 00:57:50,052 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]
2016-05-18 00:57:50,054 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533069817_0001_00_000002, launchTime=1463533070053
2016-05-18 00:57:50,054 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533069817_0001_00_000002
2016-05-18 00:57:50,054 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533069817_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533069817_0001_1_01_000000_0
2016-05-18 00:57:50,055 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533069817_0001_1_01_000000_0] started. Is using containerId: [container_1463533069817_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:50,055 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B21BEE7D04E44D3698724AE23BB1CA94, taskAttemptId=attempt_1463533069817_0001_1_01_000000_0, startTime=1463533070054, containerId=container_1463533069817_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:50 Starting to run new task attempt: attempt_1463533069817_0001_1_01_000000_0
2016-05-18 00:57:50,056 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533069817_0001_1_01_000000_0
2016-05-18 00:57:50,056 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,057 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:50,057 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39] every/(1/1) ...yplatform/ALL_UNKNOWN_ALL, VertexName: B21BEE7D04E44D3698724AE23BB1CA94, VertexParallelism: 1, TaskAttemptID:attempt_1463533069817_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BC236D77EB5F41CDB7FFEEFB70D82DCF, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C2979CB7812E467882E55FFC63C02544, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:50,057 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:50,058 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533069817_0001_1_01_000000_0
2016-05-18 00:57:50,060 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533069817_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:50,060 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533069817_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:50,060 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533069817_0001_00_000001, Exiting
2016-05-18 00:57:50,060 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533069817_0001_00_000001
2016-05-18 00:57:50,060 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533069817_0001_00_000001
2016-05-18 00:57:50,061 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:50,062 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533069817_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]
2016-05-18 00:57:50,063 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533069817_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:50,063 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533069817_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:50,063 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533069817_0001_00_000001 completed successfully
2016-05-18 00:57:50,064 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:50,066 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B21BEE7D04E44D3698724AE23BB1CA94, ordinal: 1
2016-05-18 00:57:50,066 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B21BEE7D04E44D3698724AE23BB1CA94, mem on start (mb), free: 149, total: 366, max: 672
2016-05-18 00:57:50,066 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:50,066 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:50,066 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:50,071 INFO output.MROutput (MROutput.java:initialize(421)) - C2979CB7812E467882E55FFC63C02544: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:50,071 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:50,071 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[C2979CB7812E467882E55FFC63C02544:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [BC236D77EB5F41CDB7FFEEFB70D82DCF:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:50,071 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:50,071 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:50,072 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:50,072 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[C2979CB7812E467882E55FFC63C02544:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [BC236D77EB5F41CDB7FFEEFB70D82DCF:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:50,072 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:50,072 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BC236D77EB5F41CDB7FFEEFB70D82DCF being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:50,072 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:50,072 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:50,072 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BC236D77EB5F41CDB7FFEEFB70D82DCF: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:50,072 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BC236D77EB5F41CDB7FFEEFB70D82DCF: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:50,073 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BC236D77EB5F41CDB7FFEEFB70D82DCF with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:50,074 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BC236D77EB5F41CDB7FFEEFB70D82DCF
2016-05-18 00:57:50,074 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:50,074 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533069817_0001_1_01_000000_0
2016-05-18 00:57:50,075 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BC236D77EB5F41CDB7FFEEFB70D82DCF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:50,076 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533069817_0001_1_00_000000_0_10273} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533070076, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:50,076 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BC236D77EB5F41CDB7FFEEFB70D82DCF
2016-05-18 00:57:50,076 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:50,076 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BC236D77EB5F41CDB7FFEEFB70D82DCF, wasInterrupted=false
2016-05-18 00:57:50,077 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:50,077 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:50,077 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - BC236D77EB5F41CDB7FFEEFB70D82DCF: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:50,077 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:50,078 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 6B63F16A14284C92B38085996592E145
2016-05-18 00:57:50,078 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,078 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001_wd/localmode-local-dir/output/attempt_1463533069817_0001_1_00_000000_0_10273/file.out, len=181, isLocal=true
2016-05-18 00:57:50,078 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_ALL"], id: C2979CB7812E467882E55FFC63C02544
2016-05-18 00:57:50,078 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:50,078 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:50,078 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_ALL"]
2016-05-18 00:57:50,078 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:50,079 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:50,079 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BC236D77EB5F41CDB7FFEEFB70D82DCF
2016-05-18 00:57:50,079 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BC236D77EB5F41CDB7FFEEFB70D82DCF: Shuffle Runner thread complete
2016-05-18 00:57:50,079 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 6B63F16A14284C92B38085996592E145, for 1 inputs
2016-05-18 00:57:50,080 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B21BEE7D04E44D3698724AE23BB1CA94, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:50,081 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,081 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BC236D77EB5F41CDB7FFEEFB70D82DCF: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:50,086 INFO output.MROutput (MROutput.java:close(526)) - C2979CB7812E467882E55FFC63C02544 closed
2016-05-18 00:57:50,086 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533069817_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:50,087 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533069817_0001_1_01_000000_0
2016-05-18 00:57:50,087 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B21BEE7D04E44D3698724AE23BB1CA94, mem on close (mb), free: 146, total: 366, max: 672
2016-05-18 00:57:50,087 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BC236D77EB5F41CDB7FFEEFB70D82DCF
2016-05-18 00:57:50,087 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BC236D77EB5F41CDB7FFEEFB70D82DCF: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:50,087 INFO output.MROutput (MROutput.java:close(526)) - C2979CB7812E467882E55FFC63C02544 closed
2016-05-18 00:57:50,087 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:50 Completed running task attempt: attempt_1463533069817_0001_1_01_000000_0
2016-05-18 00:57:50,088 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533069817_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:50,088 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533069817_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:50,088 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533069817_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533070080, Process_Duration=7, Process_End_Time=1463533070087, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:50,089 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533069817_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:50,089 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:50,089 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:50,089 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,089 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533069817_0001_00_000002
2016-05-18 00:57:50,089 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B21BEE7D04E44D3698724AE23BB1CA94, taskAttemptId=attempt_1463533069817_0001_1_01_000000_0, creationTime=1463533070052, allocationTime=1463533070053, startTime=1463533070054, finishTime=1463533070089, timeTaken=35, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:50,090 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533069817_0001_1_01_000000_0
2016-05-18 00:57:50,090 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:TASK_FINISHED]: vertexName=B21BEE7D04E44D3698724AE23BB1CA94, taskId=task_1463533069817_0001_1_01_000000, startTime=1463533070054, finishTime=1463533070090, timeTaken=36, status=SUCCEEDED, successfulAttemptID=attempt_1463533069817_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533070080, Process_Duration=7, Process_End_Time=1463533070087, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:50,090 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:50,090 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94]
2016-05-18 00:57:50,091 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:VERTEX_FINISHED]: vertexName=B21BEE7D04E44D3698724AE23BB1CA94, vertexId=vertex_1463533069817_0001_1_01, initRequestedTime=1463533069992, initedTime=1463533069996, startRequestedTime=1463533070002, startedTime=1463533070002, finishTime=1463533070091, timeTaken=89, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533070080, Process_Duration=7, Process_End_Time=1463533070087, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533070054, firstTasksToStart=[ task_1463533069817_0001_1_01_000000 ], lastTaskFinishTime=1463533070089, lastTasksToFinish=[ task_1463533069817_0001_1_01_000000 ], minTaskDuration=35, maxTaskDuration=35, avgTaskDuration=35.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533069817_0001_1_01_000000 ], longestDurationTasks=[ task_1463533069817_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:50,091 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:50,091 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533069817_0001_00_000002, stoppedTime=1463533070091, exitStatus=0
2016-05-18 00:57:50,091 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:50,091 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:50,091 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533069817_0001_1_00 [BC236D77EB5F41CDB7FFEEFB70D82DCF]
2016-05-18 00:57:50,092 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533069817_0001_1
2016-05-18 00:57:50,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533069817_0001_1
2016-05-18 00:57:50,092 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533069817_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:50,092 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533069817_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:50,095 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: C2979CB7812E467882E55FFC63C02544 for vertex: vertex_1463533069817_0001_1_01 [B21BEE7D04E44D3698724AE23BB1CA94], outputName: C2979CB7812E467882E55FFC63C02544
2016-05-18 00:57:50,100 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533069817_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:50,100 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533069817_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:50,100 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533069817_0001_00_000002, Exiting
2016-05-18 00:57:50,100 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533069817_0001_00_000002
2016-05-18 00:57:50,100 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533069817_0001_00_000002
2016-05-18 00:57:50,100 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533069817_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:50,100 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533069817_0001_00_000002 completed successfully
2016-05-18 00:57:50,101 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:C2979CB7812E467882E55FFC63C02544 of vertex/vertexGroup:B21BEE7D04E44D3698724AE23BB1CA94 isVertexGroupOutput:false
2016-05-18 00:57:50,102 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:50,103 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533069817_0001_1, queueSize=0
2016-05-18 00:57:50,103 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533069817_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533069817_0001_1, startTime=1463533069991, finishTime=1463533070102, timeTaken=111, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=180, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066140121, Process_Duration=9, Process_End_Time=2927066140130, Read_Duration=1, Tuples_Read=40, Tuples_Written=40, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:50,103 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533069817_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:50,103 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533069817_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:50 Completed Dag: dag_1463533069817_0001_1
2016-05-18 00:57:50,104 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533069817_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:50,104 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533069817_0001_1_post
2016-05-18 00:57:50,104 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,104 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:50,104 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:50,104 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39] every/(1/1) ...yplatform/ALL_UNKNOWN_ALL, with id=dag_1463533069817_0001_1
2016-05-18 00:57:50,104 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39] every/(1/1) ...yplatform/ALL_UNKNOWN_ALL, with id=dag_1463533069817_0001_1
2016-05-18 00:57:50,111 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...yplatform/ALL_UNKNOWN_ALL, applicationId=application_1463533069817_0001
2016-05-18 00:57:50,113 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:50,113 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:50,113 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:50,113 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:50,113 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:50,114 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:50,114 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:50,115 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:50,115 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:50,115 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:50,116 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:50,116 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:50,116 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:50,117 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:50,120 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/8375AF1868D84A798982B1961F9286E1/E9AB49AAE3094A9EAA51AD73D543FC39/.tez/application_1463533069817_0001
2016-05-18 00:57:50,120 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_ALL/_temporary
2016-05-18 00:57:50,120 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.322
2016-05-18 00:57:50,190 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_UNKNOWN_ALL
2016-05-18 00:57:50,211 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:50,212 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:50,240 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.005
2016-05-18 00:57:50,246 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 00:57:50,246 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:50,246 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:50,247 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:50,260 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:50,265 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_REPLACE
2016-05-18 00:57:50,309 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:50,309 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:50,309 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_REPLACE"]
2016-05-18 00:57:50,309 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:50,309 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:50,309 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:50,310 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...tform/ALL_UNKNOWN_REPLACE
2016-05-18 00:57:50,310 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:50,322 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:50,327 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:50,327 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:50,327 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:50,336 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001 doesn't exist and is created
2016-05-18 00:57:50,337 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:50,337 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:50,349 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:50,349 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:50,350 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001_wd
2016-05-18 00:57:50,359 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533070327_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:50,359 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:50,359 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:50,361 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:50,361 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:50,361 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:50,361 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:50,363 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:50,364 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:50,364 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:50,364 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:50,364 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:50,364 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:50,364 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:50,365 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:50,365 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:50,365 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:50,366 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:50,366 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:50,366 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:50,368 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:50,368 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:50,369 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:50,369 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001_wd/localmode-log-dir
2016-05-18 00:57:50,369 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001_wd/localmode-log-dir/history.txt.appattempt_1463533070327_0001_000000, maxErrors=10
2016-05-18 00:57:50,369 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:50,370 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533070327_0001_000000, appSubmitTime=1463533070359, launchTime=1463533070359
2016-05-18 00:57:50,374 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:50,374 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:50248
2016-05-18 00:57:50,375 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533070327
2016-05-18 00:57:50,376 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533070327_0001_000000, startTime=1463533070376
2016-05-18 00:57:50,376 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:50,449 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:50,450 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:50,450 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...tform/ALL_UNKNOWN_REPLACE, applicationId=application_1463533070327_0001, dagName=[C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E] every/(1/1) ...tform/ALL_UNKNOWN_REPLACE
2016-05-18 00:57:50,477 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E] every/(1/1) ...tform/ALL_UNKNOWN_REPLACE
2016-05-18 00:57:50,478 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:50,478 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533070327_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001_wd/localmode-log-dir/dag_1463533070327_0001_1.dot
2016-05-18 00:57:50,479 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001_wd/localmode-log-dir/dag_1463533070327_0001_1-tez-dag.pb.txt
2016-05-18 00:57:50,485 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533070327_0001_1
2016-05-18 00:57:50,485 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,485 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E] every/(1/1) ...tform/ALL_UNKNOWN_REPLACE
2016-05-18 00:57:50 Running Dag: dag_1463533070327_0001_1
2016-05-18 00:57:50,494 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533070327_0001_1, submitTime=1463533070477
2016-05-18 00:57:50,495 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001/tez-conf.pb" } size: 36913 timestamp: 1463533070000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:50,495 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,495 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:50,495 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001/tez-conf.pb" } size: 36913 timestamp: 1463533070000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:50,495 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,496 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:50,496 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:50,496 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533070327_0001_1, initTime=1463533070496
2016-05-18 00:57:50,496 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533070327_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:50,496 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:50,497 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:DAG_STARTED]: dagID=dag_1463533070327_0001_1, startTime=1463533070496
2016-05-18 00:57:50,497 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533070327_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:50,497 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: BC735F651A024D12A995A950144B2699 : {901180821695412E96885A99B5253C3F={InputName=901180821695412E96885A99B5253C3F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:50,497 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 901180821695412E96885A99B5253C3F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:50,497 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,497 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,497 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,497 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,497 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:50,498 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,498 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 901180821695412E96885A99B5253C3F on vertex vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,499 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...tform/ALL_UNKNOWN_REPLACE, applicationId=application_1463533070327_0001, dagName=[C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E] every/(1/1) ...tform/ALL_UNKNOWN_REPLACE
2016-05-18 00:57:50,500 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 901180821695412E96885A99B5253C3F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:50,501 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:50,501 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,501 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,501 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533070327_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:50,501 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9], numAdditionalOutputs=1
2016-05-18 00:57:50,501 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:50,502 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:50,502 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:50,502 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:50,501 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=F145D88319254C2EAF62C9D1568A9DDF, vertex=vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:50,502 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 901180821695412E96885A99B5253C3F on vertex vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,504 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DC7BB0C90B0D44BCA179EBAF466F0DA9:F145D88319254C2EAF62C9D1568A9DDF using oldmapred API
2016-05-18 00:57:50,504 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=F145D88319254C2EAF62C9D1568A9DDF, vertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:50,505 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, vertexId=vertex_1463533070327_0001_1_01, initRequestedTime=1463533070498, initedTime=1463533070501, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:50,505 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:50,506 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:50,506 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {901180821695412E96885A99B5253C3F=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:50,506 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] parallelism set to 1
2016-05-18 00:57:50,507 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:50,508 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BC735F651A024D12A995A950144B2699, vertexId=vertex_1463533070327_0001_1_00, initRequestedTime=1463533070497, initedTime=1463533070508, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:50,508 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] with distanceFromRoot: 0
2016-05-18 00:57:50,508 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:50,508 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533070327_0001_1_00, startRequestedTime=1463533070498, startedTime=1463533070508
2016-05-18 00:57:50,509 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533070327_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:50,509 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:50,509 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533070327_0001_1_00 for vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] numStartedSources: 1 numSources: 1
2016-05-18 00:57:50,509 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] with distanceFromRoot: 1
2016-05-18 00:57:50,509 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in BC735F651A024D12A995A950144B2699
2016-05-18 00:57:50,509 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,510 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533070327_0001_1_01, startRequestedTime=1463533070509, startedTime=1463533070509
2016-05-18 00:57:50,510 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:50,510 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DC7BB0C90B0D44BCA179EBAF466F0DA9 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:50,510 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:TASK_STARTED]: vertexName=BC735F651A024D12A995A950144B2699, taskId=task_1463533070327_0001_1_00_000000, scheduledTime=1463533070510, launchTime=1463533070510
2016-05-18 00:57:50,511 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,511 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BC735F651A024D12A995A950144B2699 in vertex: DC7BB0C90B0D44BCA179EBAF466F0DA9 numBipartiteSourceTasks: 1
2016-05-18 00:57:50,511 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:50,515 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533070327_0001_00_000001, launchTime=1463533070515
2016-05-18 00:57:50,516 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533070327_0001_00_000001
2016-05-18 00:57:50,516 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533070327_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533070327_0001_1_00_000000_0
2016-05-18 00:57:50,517 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533070327_0001_1_00_000000_0] started. Is using containerId: [container_1463533070327_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:50,517 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BC735F651A024D12A995A950144B2699, taskAttemptId=attempt_1463533070327_0001_1_00_000000_0, startTime=1463533070516, containerId=container_1463533070327_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:50 Starting to run new task attempt: attempt_1463533070327_0001_1_00_000000_0
2016-05-18 00:57:50,517 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533070327_0001_1_00_000000_0
2016-05-18 00:57:50,517 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,518 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:50,518 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E] every/(1/1) ...tform/ALL_UNKNOWN_REPLACE, VertexName: BC735F651A024D12A995A950144B2699, VertexParallelism: 1, TaskAttemptID:attempt_1463533070327_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=901180821695412E96885A99B5253C3F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:50,518 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:50,519 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533070327_0001_1_00_000000_0
2016-05-18 00:57:50,519 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533070327_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:50,520 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533070327_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:50,523 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:50,523 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533070327_0001, with dag id: dag_1463533070327_0001_1
2016-05-18 00:57:50,525 INFO input.MRInput (MRInput.java:initialize(440)) - 901180821695412E96885A99B5253C3F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:50,525 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 901180821695412E96885A99B5253C3F
2016-05-18 00:57:50,528 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BC735F651A024D12A995A950144B2699, ordinal: 0
2016-05-18 00:57:50,528 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BC735F651A024D12A995A950144B2699, mem on start (mb), free: 253, total: 366, max: 672
2016-05-18 00:57:50,528 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:50,528 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:50,528 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:50,529 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:50,529 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[901180821695412E96885A99B5253C3F:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [DC7BB0C90B0D44BCA179EBAF466F0DA9:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:50,529 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:50,530 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:50,530 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:50,530 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[901180821695412E96885A99B5253C3F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [DC7BB0C90B0D44BCA179EBAF466F0DA9:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:50,531 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:50,531 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 901180821695412E96885A99B5253C3F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:50,531 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:50,531 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:50,531 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 901180821695412E96885A99B5253C3F
2016-05-18 00:57:50,532 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:50,532 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533070327_0001_1_00_000000_0
2016-05-18 00:57:50,533 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:50,533 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 901180821695412E96885A99B5253C3F initialized RecordReader from event
2016-05-18 00:57:50,534 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:50,535 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 901180821695412E96885A99B5253C3F
2016-05-18 00:57:50,535 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: BD0D6300BFFC4C648C8C83B42021ED19
2016-05-18 00:57:50,535 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] BD0D6300BFFC4C648C8C83B42021ED19
2016-05-18 00:57:50,536 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,538 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,538 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DC7BB0C90B0D44BCA179EBAF466F0DA9 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@11473ebd, comparator=cascading.tuple.hadoop.util.TupleComparator@5c67f327, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:50,538 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DC7BB0C90B0D44BCA179EBAF466F0DA9: , UsingHashComparator=false
2016-05-18 00:57:50,552 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:50,552 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:50,552 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DC7BB0C90B0D44BCA179EBAF466F0DA9: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:50,552 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DC7BB0C90B0D44BCA179EBAF466F0DA9: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:50,553 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:50,553 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BC735F651A024D12A995A950144B2699, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:50,556 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533070327_0001_1_00_000000_0
2016-05-18 00:57:50,556 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BC735F651A024D12A995A950144B2699, mem on close (mb), free: 152, total: 366, max: 672
2016-05-18 00:57:50,557 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DC7BB0C90B0D44BCA179EBAF466F0DA9: Starting flush of map output
2016-05-18 00:57:50,557 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DC7BB0C90B0D44BCA179EBAF466F0DA9: Span0.length = 20, perItem = 6
2016-05-18 00:57:50,557 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DC7BB0C90B0D44BCA179EBAF466F0DA9: done sorting span=0, length=20, time=0
2016-05-18 00:57:50,557 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DC7BB0C90B0D44BCA179EBAF466F0DA9: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:50,558 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DC7BB0C90B0D44BCA179EBAF466F0DA9: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001_wd/localmode-local-dir/output/attempt_1463533070327_0001_1_00_000000_0_10279_0/file.out
2016-05-18 00:57:50,561 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:50 Completed running task attempt: attempt_1463533070327_0001_1_00_000000_0
2016-05-18 00:57:50,562 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533070327_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:50,562 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533070327_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:50,562 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533070327_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533070553, Process_Duration=3, Process_End_Time=1463533070556, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:50,563 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533070327_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:50,563 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:50,563 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:50,563 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,563 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533070327_0001_00_000001
2016-05-18 00:57:50,563 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BC735F651A024D12A995A950144B2699, taskAttemptId=attempt_1463533070327_0001_1_00_000000_0, creationTime=1463533070510, allocationTime=1463533070512, startTime=1463533070516, finishTime=1463533070563, timeTaken=47, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:50,564 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533070327_0001_1_00_000000_0
2016-05-18 00:57:50,564 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:TASK_FINISHED]: vertexName=BC735F651A024D12A995A950144B2699, taskId=task_1463533070327_0001_1_00_000000, startTime=1463533070516, finishTime=1463533070564, timeTaken=48, status=SUCCEEDED, successfulAttemptID=attempt_1463533070327_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533070553, Process_Duration=3, Process_End_Time=1463533070556, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:50,564 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:50,565 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699]
2016-05-18 00:57:50,565 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:VERTEX_FINISHED]: vertexName=BC735F651A024D12A995A950144B2699, vertexId=vertex_1463533070327_0001_1_00, initRequestedTime=1463533070497, initedTime=1463533070508, startRequestedTime=1463533070498, startedTime=1463533070508, finishTime=1463533070565, timeTaken=57, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533070553, Process_Duration=3, Process_End_Time=1463533070556, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533070516, firstTasksToStart=[ task_1463533070327_0001_1_00_000000 ], lastTaskFinishTime=1463533070563, lastTasksToFinish=[ task_1463533070327_0001_1_00_000000 ], minTaskDuration=47, maxTaskDuration=47, avgTaskDuration=47.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533070327_0001_1_00_000000 ], longestDurationTasks=[ task_1463533070327_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:50,565 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:50,565 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] attempt: attempt_1463533070327_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:50,566 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DC7BB0C90B0D44BCA179EBAF466F0DA9
2016-05-18 00:57:50,566 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533070327_0001_00_000001, stoppedTime=1463533070566, exitStatus=0
2016-05-18 00:57:50,566 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BC735F651A024D12A995A950144B2699, destinationVertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:50,566 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,566 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533070327_0001_1_00 [BC735F651A024D12A995A950144B2699] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:50,566 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:50,566 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:TASK_STARTED]: vertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, taskId=task_1463533070327_0001_1_01_000000, scheduledTime=1463533070566, launchTime=1463533070566
2016-05-18 00:57:50,567 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,567 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533070327_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:50,568 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533070327_0001_00_000002, launchTime=1463533070568
2016-05-18 00:57:50,569 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533070327_0001_00_000002
2016-05-18 00:57:50,569 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533070327_0001_1_01_000000_0] started. Is using containerId: [container_1463533070327_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:50,569 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, taskAttemptId=attempt_1463533070327_0001_1_01_000000_0, startTime=1463533070569, containerId=container_1463533070327_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:50,570 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533070327_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533070327_0001_1_01_000000_0
2016-05-18 00:57:50 Starting to run new task attempt: attempt_1463533070327_0001_1_01_000000_0
2016-05-18 00:57:50,570 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533070327_0001_1_01_000000_0
2016-05-18 00:57:50,570 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,570 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:50,570 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E] every/(1/1) ...tform/ALL_UNKNOWN_REPLACE, VertexName: DC7BB0C90B0D44BCA179EBAF466F0DA9, VertexParallelism: 1, TaskAttemptID:attempt_1463533070327_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BC735F651A024D12A995A950144B2699, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=F145D88319254C2EAF62C9D1568A9DDF, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:50,570 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:50,571 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533070327_0001_1_01_000000_0
2016-05-18 00:57:50,571 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533070327_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,571 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533070327_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:50,574 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533070327_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:50,574 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533070327_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:50,574 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533070327_0001_00_000001, Exiting
2016-05-18 00:57:50,574 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533070327_0001_00_000001
2016-05-18 00:57:50,574 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533070327_0001_00_000001
2016-05-18 00:57:50,574 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:50,576 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:50,577 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533070327_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:50,577 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533070327_0001_00_000001 completed successfully
2016-05-18 00:57:50,579 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DC7BB0C90B0D44BCA179EBAF466F0DA9, ordinal: 1
2016-05-18 00:57:50,579 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DC7BB0C90B0D44BCA179EBAF466F0DA9, mem on start (mb), free: 146, total: 366, max: 672
2016-05-18 00:57:50,579 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:50,579 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:50,579 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:50,581 INFO output.MROutput (MROutput.java:initialize(421)) - F145D88319254C2EAF62C9D1568A9DDF: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:50,581 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:50,581 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F145D88319254C2EAF62C9D1568A9DDF:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [BC735F651A024D12A995A950144B2699:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:50,582 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:50,582 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:50,582 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:50,582 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F145D88319254C2EAF62C9D1568A9DDF:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [BC735F651A024D12A995A950144B2699:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:50,582 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:50,582 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BC735F651A024D12A995A950144B2699 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:50,582 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BC735F651A024D12A995A950144B2699: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:50,582 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:50,582 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BC735F651A024D12A995A950144B2699: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:50,582 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:50,584 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BC735F651A024D12A995A950144B2699 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:50,584 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BC735F651A024D12A995A950144B2699
2016-05-18 00:57:50,585 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:50,585 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533070327_0001_1_01_000000_0
2016-05-18 00:57:50,585 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BC735F651A024D12A995A950144B2699: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:50,586 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533070327_0001_1_00_000000_0_10279} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533070586, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:50,586 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BC735F651A024D12A995A950144B2699
2016-05-18 00:57:50,586 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:50,587 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BC735F651A024D12A995A950144B2699, wasInterrupted=false
2016-05-18 00:57:50,587 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:50,587 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:50,587 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:50,588 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: BD0D6300BFFC4C648C8C83B42021ED19
2016-05-18 00:57:50,588 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_REPLACE"], id: F145D88319254C2EAF62C9D1568A9DDF
2016-05-18 00:57:50,588 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_REPLACE"]
2016-05-18 00:57:50,588 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] BD0D6300BFFC4C648C8C83B42021ED19, for 1 inputs
2016-05-18 00:57:50,588 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,589 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001_wd/localmode-local-dir/output/attempt_1463533070327_0001_1_00_000000_0_10279/file.out, len=181, isLocal=true
2016-05-18 00:57:50,589 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:50,589 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:50,589 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:50,589 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:50,589 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BC735F651A024D12A995A950144B2699
2016-05-18 00:57:50,589 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BC735F651A024D12A995A950144B2699: Shuffle Runner thread complete
2016-05-18 00:57:50,589 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DC7BB0C90B0D44BCA179EBAF466F0DA9, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:57:50,590 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,591 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BC735F651A024D12A995A950144B2699: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:50,593 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:50,593 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:50,594 INFO output.MROutput (MROutput.java:close(526)) - F145D88319254C2EAF62C9D1568A9DDF closed
2016-05-18 00:57:50,594 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533070327_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:50,594 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533070327_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:50,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533070327_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533070590, Process_Duration=4, Process_End_Time=1463533070594, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:50,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:50,595 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DC7BB0C90B0D44BCA179EBAF466F0DA9, mem on close (mb), free: 143, total: 366, max: 672
2016-05-18 00:57:50,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=DC7BB0C90B0D44BCA179EBAF466F0DA9, index=1, interruptedStatus=false
2016-05-18 00:57:50,595 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BC735F651A024D12A995A950144B2699
2016-05-18 00:57:50,595 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BC735F651A024D12A995A950144B2699: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:50,596 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=DC7BB0C90B0D44BCA179EBAF466F0DA9, sourceVertex=BC735F651A024D12A995A950144B2699, interruptedStatus=true
2016-05-18 00:57:50,596 INFO output.MROutput (MROutput.java:close(526)) - F145D88319254C2EAF62C9D1568A9DDF closed
2016-05-18 00:57:50,596 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=DC7BB0C90B0D44BCA179EBAF466F0DA9, sourceVertex=F145D88319254C2EAF62C9D1568A9DDF, interruptedStatus=true
2016-05-18 00:57:50,596 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533070327_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], containerShutdownRequested=false}
2016-05-18 00:57:50,596 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], containerShutdownRequested=false}
2016-05-18 00:57:50,597 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533070327_0001_00_000002
2016-05-18 00:57:50,597 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533070327_0001_00_000002
2016-05-18 00:57:50,597 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533070327_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]'}
2016-05-18 00:57:50,597 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533070327_0001_00_000002 completed but with errors
2016-05-18 00:57:50,597 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, taskAttemptId=attempt_1463533070327_0001_1_01_000000_0, creationTime=1463533070566, allocationTime=1463533070568, startTime=1463533070569, finishTime=1463533070597, timeTaken=28, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533070327_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533070590, Process_Duration=4, Process_End_Time=1463533070594, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:50,597 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533070327_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
2016-05-18 00:57:50,597 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533070327_0001_00_000002, stoppedTime=1463533070597, exitStatus=1
2016-05-18 00:57:50,597 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533070327_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:50,597 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:TASK_FINISHED]: vertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, taskId=task_1463533070327_0001_1_01_000000, startTime=1463533070569, finishTime=1463533070597, timeTaken=28, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533070327_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:50,598 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] because task failed: task_1463533070327_0001_1_01_000000
2016-05-18 00:57:50,598 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:50,598 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:50,598 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:50,598 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,598 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=F145D88319254C2EAF62C9D1568A9DDF, vertexId=vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,599 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:VERTEX_FINISHED]: vertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, vertexId=vertex_1463533070327_0001_1_01, initRequestedTime=1463533070498, initedTime=1463533070501, startRequestedTime=1463533070509, startedTime=1463533070509, finishTime=1463533070599, timeTaken=90, status=FAILED, diagnostics=Task failed, taskId=task_1463533070327_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533070327_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533070569, firstTasksToStart=[ task_1463533070327_0001_1_01_000000 ], lastTaskFinishTime=1463533070597, lastTasksToFinish=[ task_1463533070327_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:50,599 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:50,599 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:50,599 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:50,599 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:50,600 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533070327_0001_1, queueSize=0
2016-05-18 00:57:50,600 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070327_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533070327_0001_1, startTime=1463533070496, finishTime=1463533070599, timeTaken=103, status=FAILED, diagnostics=Vertex failed, vertexName=DC7BB0C90B0D44BCA179EBAF466F0DA9, vertexId=vertex_1463533070327_0001_1_01, diagnostics=[Task failed, taskId=task_1463533070327_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533070327_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:UNKNOWN]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=170, AM_GC_TIME_MILLIS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533070553, Process_Duration=3, Process_End_Time=1463533070556, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:50,600 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533070327_0001_1_01 [DC7BB0C90B0D44BCA179EBAF466F0DA9]
2016-05-18 00:57:50,600 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533070327_0001_1 finished with state: FAILED
2016-05-18 00:57:50,600 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533070327_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:50 Completed Dag: dag_1463533070327_0001_1
2016-05-18 00:57:50,600 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533070327_0001_1, dagState=FAILED
2016-05-18 00:57:50,601 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533070327_0001_1_post
2016-05-18 00:57:50,601 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 2
2016-05-18 00:57:50,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:50,601 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533070327_0001_1_01_000000_0 failed: true container: container_1463533070327_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:50,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E] every/(1/1) ...tform/ALL_UNKNOWN_REPLACE, with id=dag_1463533070327_0001_1
2016-05-18 00:57:50,601 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E] every/(1/1) ...tform/ALL_UNKNOWN_REPLACE, with id=dag_1463533070327_0001_1
2016-05-18 00:57:50,607 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...tform/ALL_UNKNOWN_REPLACE, applicationId=application_1463533070327_0001
2016-05-18 00:57:50,608 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:50,608 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:50,608 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:50,608 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:50,608 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:50,609 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:50,609 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:50,609 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:50,609 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:50,609 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:50,609 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:50,610 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:50,610 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:50,611 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:50,613 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/C3AC859BAE2A45578E1CAB2BB543A74D/CB1C9F4F51324DC4AC07DD5008053E0E/.tez/application_1463533070327_0001
2016-05-18 00:57:50,613 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:50,614 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...tform/ALL_UNKNOWN_REPLACE
2016-05-18 00:57:50,614 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:50,614 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_REPLACE/_temporary
2016-05-18 00:57:50,614 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.305
2016-05-18 00:57:50,615 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_UNKNOWN_REPLACE
2016-05-18 00:57:50,635 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:50,636 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:50,664 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:50,668 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.010
2016-05-18 00:57:50,668 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:50,668 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:50,668 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:50,682 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:50,686 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_SWAP
2016-05-18 00:57:50,728 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:50,728 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:50,728 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_SWAP"]
2016-05-18 00:57:50,728 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:50,728 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:50,728 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:50,729 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...platform/ALL_UNKNOWN_SWAP
2016-05-18 00:57:50,729 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:50,740 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:50,744 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:50,744 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:50,744 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:50,753 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001 doesn't exist and is created
2016-05-18 00:57:50,776 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:50,776 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:50,777 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001_wd
2016-05-18 00:57:50,786 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533070744_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:50,786 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:50,786 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:50,788 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:50,788 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:50,788 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:50,788 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:50,788 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:50,790 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:50,790 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:50,790 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:50,792 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:50,792 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:50,792 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:50,792 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001_wd/localmode-log-dir
2016-05-18 00:57:50,793 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001_wd/localmode-log-dir/history.txt.appattempt_1463533070744_0001_000000, maxErrors=10
2016-05-18 00:57:50,793 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:50,793 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533070744_0001_000000, appSubmitTime=1463533070786, launchTime=1463533070786
2016-05-18 00:57:50,797 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:50,800 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:55102
2016-05-18 00:57:50,800 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533070744
2016-05-18 00:57:50,801 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533070744_0001_000000, startTime=1463533070801
2016-05-18 00:57:50,801 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:50,869 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:50,869 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:50,877 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:50,877 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:50,877 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...platform/ALL_UNKNOWN_SWAP, applicationId=application_1463533070744_0001, dagName=[FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C] every/(1/1) ...platform/ALL_UNKNOWN_SWAP
2016-05-18 00:57:50,892 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C] every/(1/1) ...platform/ALL_UNKNOWN_SWAP
2016-05-18 00:57:50,892 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:50,893 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533070744_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001_wd/localmode-log-dir/dag_1463533070744_0001_1.dot
2016-05-18 00:57:50,893 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001_wd/localmode-log-dir/dag_1463533070744_0001_1-tez-dag.pb.txt
2016-05-18 00:57:50,900 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533070744_0001_1
2016-05-18 00:57:50,900 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,900 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C] every/(1/1) ...platform/ALL_UNKNOWN_SWAP
2016-05-18 00:57:50 Running Dag: dag_1463533070744_0001_1
2016-05-18 00:57:50,908 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533070744_0001_1, submitTime=1463533070892
2016-05-18 00:57:50,909 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001/tez-conf.pb" } size: 36913 timestamp: 1463533070000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:50,909 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,909 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:50,910 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001/tez-conf.pb" } size: 36913 timestamp: 1463533070000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:50,910 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]
2016-05-18 00:57:50,910 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:50,910 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:50,910 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533070744_0001_1, initTime=1463533070910
2016-05-18 00:57:50,910 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533070744_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:50,911 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:50,911 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:DAG_STARTED]: dagID=dag_1463533070744_0001_1, startTime=1463533070911
2016-05-18 00:57:50,911 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533070744_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:50,911 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: ABCFC83C9AF147DEA084F8D993F4BC9C : {B50E3B37819C47828F03F6AB62899A04={InputName=B50E3B37819C47828F03F6AB62899A04}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:50,911 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: B50E3B37819C47828F03F6AB62899A04, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:50,911 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,911 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,911 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,911 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,912 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...platform/ALL_UNKNOWN_SWAP, applicationId=application_1463533070744_0001, dagName=[FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C] every/(1/1) ...platform/ALL_UNKNOWN_SWAP
2016-05-18 00:57:50,912 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:50,912 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: B50E3B37819C47828F03F6AB62899A04 on vertex vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,912 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]
2016-05-18 00:57:50,915 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input B50E3B37819C47828F03F6AB62899A04 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:50,915 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:50,915 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]
2016-05-18 00:57:50,916 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]
2016-05-18 00:57:50,916 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533070744_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:50,916 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A], numAdditionalOutputs=1
2016-05-18 00:57:50,916 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=DBB8296F1DA54ADCA65B08320097550E, vertex=vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:50,917 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:50,917 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:50,917 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:50,918 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:50,918 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: B50E3B37819C47828F03F6AB62899A04 on vertex vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,919 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for EC4B65F847404BD89201506981CE010A:DBB8296F1DA54ADCA65B08320097550E using oldmapred API
2016-05-18 00:57:50,919 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=DBB8296F1DA54ADCA65B08320097550E, vertexName=EC4B65F847404BD89201506981CE010A, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:50,920 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:VERTEX_INITIALIZED]: vertexName=EC4B65F847404BD89201506981CE010A, vertexId=vertex_1463533070744_0001_1_01, initRequestedTime=1463533070912, initedTime=1463533070916, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:50,920 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:50,920 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:50,921 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {B50E3B37819C47828F03F6AB62899A04=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:50,921 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] parallelism set to 1
2016-05-18 00:57:50,921 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:50,921 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ABCFC83C9AF147DEA084F8D993F4BC9C, vertexId=vertex_1463533070744_0001_1_00, initRequestedTime=1463533070911, initedTime=1463533070921, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:50,921 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] with distanceFromRoot: 0
2016-05-18 00:57:50,921 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:50,922 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533070744_0001_1_00, startRequestedTime=1463533070912, startedTime=1463533070922
2016-05-18 00:57:50,922 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533070744_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:50,922 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:50,922 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533070744_0001_1_00 for vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A] numStartedSources: 1 numSources: 1
2016-05-18 00:57:50,922 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A] with distanceFromRoot: 1
2016-05-18 00:57:50,922 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533070744_0001_1_01, startRequestedTime=1463533070922, startedTime=1463533070922
2016-05-18 00:57:50,922 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:50,923 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in ABCFC83C9AF147DEA084F8D993F4BC9C
2016-05-18 00:57:50,923 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,924 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:TASK_STARTED]: vertexName=ABCFC83C9AF147DEA084F8D993F4BC9C, taskId=task_1463533070744_0001_1_00_000000, scheduledTime=1463533070924, launchTime=1463533070924
2016-05-18 00:57:50,924 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: EC4B65F847404BD89201506981CE010A with 0 source tasks and 1 pending tasks
2016-05-18 00:57:50,924 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,925 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: ABCFC83C9AF147DEA084F8D993F4BC9C in vertex: EC4B65F847404BD89201506981CE010A numBipartiteSourceTasks: 1
2016-05-18 00:57:50,925 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:50,927 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533070744_0001_00_000001, launchTime=1463533070926
2016-05-18 00:57:50,927 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533070744_0001_00_000001
2016-05-18 00:57:50,927 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533070744_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533070744_0001_1_00_000000_0
2016-05-18 00:57:50,928 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533070744_0001_1_00_000000_0] started. Is using containerId: [container_1463533070744_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:50,928 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ABCFC83C9AF147DEA084F8D993F4BC9C, taskAttemptId=attempt_1463533070744_0001_1_00_000000_0, startTime=1463533070928, containerId=container_1463533070744_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:50 Starting to run new task attempt: attempt_1463533070744_0001_1_00_000000_0
2016-05-18 00:57:50,928 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533070744_0001_1_00_000000_0
2016-05-18 00:57:50,929 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,944 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:50,944 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C] every/(1/1) ...platform/ALL_UNKNOWN_SWAP, VertexName: ABCFC83C9AF147DEA084F8D993F4BC9C, VertexParallelism: 1, TaskAttemptID:attempt_1463533070744_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=B50E3B37819C47828F03F6AB62899A04, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=EC4B65F847404BD89201506981CE010A, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:50,944 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:50,944 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533070744_0001, with dag id: dag_1463533070744_0001_1
2016-05-18 00:57:50,944 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533070744_0001_1_00_000000_0
2016-05-18 00:57:50,946 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533070744_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:50,947 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533070744_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:50,948 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:50,952 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ABCFC83C9AF147DEA084F8D993F4BC9C, ordinal: 0
2016-05-18 00:57:50,952 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ABCFC83C9AF147DEA084F8D993F4BC9C, mem on start (mb), free: 271, total: 367, max: 672
2016-05-18 00:57:50,952 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:50,952 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:50,952 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:50,954 INFO input.MRInput (MRInput.java:initialize(440)) - B50E3B37819C47828F03F6AB62899A04 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:50,954 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: B50E3B37819C47828F03F6AB62899A04
2016-05-18 00:57:50,955 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:50,955 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[EC4B65F847404BD89201506981CE010A:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [B50E3B37819C47828F03F6AB62899A04:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 00:57:50,955 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:50,956 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:50,956 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:50,957 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[EC4B65F847404BD89201506981CE010A:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [B50E3B37819C47828F03F6AB62899A04:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 00:57:50,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:50,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: B50E3B37819C47828F03F6AB62899A04 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:50,957 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: B50E3B37819C47828F03F6AB62899A04
2016-05-18 00:57:50,958 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:50,958 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:50,958 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:50,958 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533070744_0001_1_00_000000_0
2016-05-18 00:57:50,959 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:50,959 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - B50E3B37819C47828F03F6AB62899A04 initialized RecordReader from event
2016-05-18 00:57:50,961 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:50,961 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: B50E3B37819C47828F03F6AB62899A04
2016-05-18 00:57:50,961 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: BB864A811D6D4E628C768795B08DC9A7
2016-05-18 00:57:50,961 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] BB864A811D6D4E628C768795B08DC9A7
2016-05-18 00:57:50,962 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,964 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:50,964 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - EC4B65F847404BD89201506981CE010A using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@4ee5ebba, comparator=cascading.tuple.hadoop.util.TupleComparator@58d728cd, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:50,964 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for EC4B65F847404BD89201506981CE010A: , UsingHashComparator=false
2016-05-18 00:57:50,977 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:50,978 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:50,978 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for EC4B65F847404BD89201506981CE010A: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:50,978 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - EC4B65F847404BD89201506981CE010A: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:50,978 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:50,978 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ABCFC83C9AF147DEA084F8D993F4BC9C, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:50,980 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533070744_0001_1_00_000000_0
2016-05-18 00:57:50,980 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ABCFC83C9AF147DEA084F8D993F4BC9C, mem on close (mb), free: 168, total: 367, max: 672
2016-05-18 00:57:50,981 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - EC4B65F847404BD89201506981CE010A: Starting flush of map output
2016-05-18 00:57:50,981 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - EC4B65F847404BD89201506981CE010A: Span0.length = 20, perItem = 6
2016-05-18 00:57:50,981 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - EC4B65F847404BD89201506981CE010A: done sorting span=0, length=20, time=0
2016-05-18 00:57:50,981 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - EC4B65F847404BD89201506981CE010A: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:50,982 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - EC4B65F847404BD89201506981CE010A: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001_wd/localmode-local-dir/output/attempt_1463533070744_0001_1_00_000000_0_10284_0/file.out
2016-05-18 00:57:50,985 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:50 Completed running task attempt: attempt_1463533070744_0001_1_00_000000_0
2016-05-18 00:57:50,985 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533070744_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:50,986 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533070744_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:50,986 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533070744_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533070978, Process_Duration=2, Process_End_Time=1463533070980, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:50,986 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533070744_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:50,986 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:50,986 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ABCFC83C9AF147DEA084F8D993F4BC9C, taskAttemptId=attempt_1463533070744_0001_1_00_000000_0, creationTime=1463533070924, allocationTime=1463533070926, startTime=1463533070928, finishTime=1463533070986, timeTaken=58, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:50,987 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533070744_0001_1_00_000000_0
2016-05-18 00:57:50,987 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:TASK_FINISHED]: vertexName=ABCFC83C9AF147DEA084F8D993F4BC9C, taskId=task_1463533070744_0001_1_00_000000, startTime=1463533070928, finishTime=1463533070987, timeTaken=59, status=SUCCEEDED, successfulAttemptID=attempt_1463533070744_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533070978, Process_Duration=2, Process_End_Time=1463533070980, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:50,986 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:50,987 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,987 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:50,987 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:50,987 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533070744_0001_00_000001
2016-05-18 00:57:50,988 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:VERTEX_FINISHED]: vertexName=ABCFC83C9AF147DEA084F8D993F4BC9C, vertexId=vertex_1463533070744_0001_1_00, initRequestedTime=1463533070911, initedTime=1463533070921, startRequestedTime=1463533070912, startedTime=1463533070922, finishTime=1463533070988, timeTaken=66, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533070978, Process_Duration=2, Process_End_Time=1463533070980, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533070928, firstTasksToStart=[ task_1463533070744_0001_1_00_000000 ], lastTaskFinishTime=1463533070986, lastTasksToFinish=[ task_1463533070744_0001_1_00_000000 ], minTaskDuration=58, maxTaskDuration=58, avgTaskDuration=58.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533070744_0001_1_00_000000 ], longestDurationTasks=[ task_1463533070744_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:50,988 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:50,988 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533070744_0001_00_000001, stoppedTime=1463533070988, exitStatus=0
2016-05-18 00:57:50,988 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A] attempt: attempt_1463533070744_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:50,989 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:50,989 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:50,989 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: EC4B65F847404BD89201506981CE010A
2016-05-18 00:57:50,989 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533070744_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:50,989 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=ABCFC83C9AF147DEA084F8D993F4BC9C, destinationVertexName=EC4B65F847404BD89201506981CE010A. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:50,989 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]
2016-05-18 00:57:50,989 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:TASK_STARTED]: vertexName=EC4B65F847404BD89201506981CE010A, taskId=task_1463533070744_0001_1_01_000000, scheduledTime=1463533070989, launchTime=1463533070989
2016-05-18 00:57:50,990 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]
2016-05-18 00:57:50,991 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533070744_0001_00_000002, launchTime=1463533070991
2016-05-18 00:57:50,991 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533070744_0001_00_000002
2016-05-18 00:57:50,992 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533070744_0001_1_01_000000_0] started. Is using containerId: [container_1463533070744_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:50,992 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=EC4B65F847404BD89201506981CE010A, taskAttemptId=attempt_1463533070744_0001_1_01_000000_0, startTime=1463533070992, containerId=container_1463533070744_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:50,992 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533070744_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533070744_0001_1_01_000000_0
2016-05-18 00:57:50 Starting to run new task attempt: attempt_1463533070744_0001_1_01_000000_0
2016-05-18 00:57:50,994 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533070744_0001_1_01_000000_0
2016-05-18 00:57:50,994 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:50,994 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:50,994 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C] every/(1/1) ...platform/ALL_UNKNOWN_SWAP, VertexName: EC4B65F847404BD89201506981CE010A, VertexParallelism: 1, TaskAttemptID:attempt_1463533070744_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ABCFC83C9AF147DEA084F8D993F4BC9C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=DBB8296F1DA54ADCA65B08320097550E, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:50,994 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:50,995 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533070744_0001_1_01_000000_0
2016-05-18 00:57:50,997 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:50,997 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533070744_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]
2016-05-18 00:57:50,998 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533070744_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:50,998 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:50,998 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533070744_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:50,998 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533070744_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:50,998 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533070744_0001_00_000001, Exiting
2016-05-18 00:57:50,998 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533070744_0001_00_000001
2016-05-18 00:57:50,998 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533070744_0001_00_000001
2016-05-18 00:57:50,999 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533070744_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:50,999 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533070744_0001_00_000001 completed successfully
2016-05-18 00:57:51,001 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: EC4B65F847404BD89201506981CE010A, ordinal: 1
2016-05-18 00:57:51,001 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EC4B65F847404BD89201506981CE010A, mem on start (mb), free: 163, total: 367, max: 672
2016-05-18 00:57:51,001 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:51,002 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:51,002 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:51,006 INFO output.MROutput (MROutput.java:initialize(421)) - DBB8296F1DA54ADCA65B08320097550E: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:51,007 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:51,007 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DBB8296F1DA54ADCA65B08320097550E:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [ABCFC83C9AF147DEA084F8D993F4BC9C:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:51,007 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:51,007 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:51,007 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:51,007 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DBB8296F1DA54ADCA65B08320097550E:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [ABCFC83C9AF147DEA084F8D993F4BC9C:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:51,007 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:51,008 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: ABCFC83C9AF147DEA084F8D993F4BC9C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:51,008 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - ABCFC83C9AF147DEA084F8D993F4BC9C: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:51,008 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:51,008 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:51,008 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - ABCFC83C9AF147DEA084F8D993F4BC9C: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:51,009 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: ABCFC83C9AF147DEA084F8D993F4BC9C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:51,010 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: ABCFC83C9AF147DEA084F8D993F4BC9C
2016-05-18 00:57:51,010 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:51,010 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533070744_0001_1_01_000000_0
2016-05-18 00:57:51,010 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - ABCFC83C9AF147DEA084F8D993F4BC9C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:51,011 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533070744_0001_1_00_000000_0_10284} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533071011, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:51,011 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : ABCFC83C9AF147DEA084F8D993F4BC9C
2016-05-18 00:57:51,012 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:51,012 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: ABCFC83C9AF147DEA084F8D993F4BC9C, wasInterrupted=false
2016-05-18 00:57:51,012 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:51,013 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:51,013 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:51,013 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: BB864A811D6D4E628C768795B08DC9A7
2016-05-18 00:57:51,013 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_SWAP"], id: DBB8296F1DA54ADCA65B08320097550E
2016-05-18 00:57:51,013 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_SWAP"]
2016-05-18 00:57:51,013 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] BB864A811D6D4E628C768795B08DC9A7, for 1 inputs
2016-05-18 00:57:51,014 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:51,014 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001_wd/localmode-local-dir/output/attempt_1463533070744_0001_1_00_000000_0_10284/file.out, len=181, isLocal=true
2016-05-18 00:57:51,014 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:51,014 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:51,014 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:51,014 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:51,014 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : ABCFC83C9AF147DEA084F8D993F4BC9C
2016-05-18 00:57:51,015 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - ABCFC83C9AF147DEA084F8D993F4BC9C: Shuffle Runner thread complete
2016-05-18 00:57:51,015 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: EC4B65F847404BD89201506981CE010A, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:51,015 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:51,016 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - ABCFC83C9AF147DEA084F8D993F4BC9C: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:51,020 INFO output.MROutput (MROutput.java:close(526)) - DBB8296F1DA54ADCA65B08320097550E closed
2016-05-18 00:57:51,020 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533070744_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:51,021 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533070744_0001_1_01_000000_0
2016-05-18 00:57:51,021 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: EC4B65F847404BD89201506981CE010A, mem on close (mb), free: 160, total: 367, max: 672
2016-05-18 00:57:51,021 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: ABCFC83C9AF147DEA084F8D993F4BC9C
2016-05-18 00:57:51,021 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - ABCFC83C9AF147DEA084F8D993F4BC9C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:51,021 INFO output.MROutput (MROutput.java:close(526)) - DBB8296F1DA54ADCA65B08320097550E closed
2016-05-18 00:57:51,021 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:51 Completed running task attempt: attempt_1463533070744_0001_1_01_000000_0
2016-05-18 00:57:51,021 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533070744_0001_1_01_000000_0, askedToStop=false
2016-05-18 00:57:51,021 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533070744_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:51,022 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533070744_0001_1_01_000000_0: Counters: 36 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=6, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533071015, Process_Duration=6, Process_End_Time=1463533071021, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=20]]
2016-05-18 00:57:51,022 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533070744_0001_1_01_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:51,022 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:51,023 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:51,023 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:51,023 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=EC4B65F847404BD89201506981CE010A, taskAttemptId=attempt_1463533070744_0001_1_01_000000_0, creationTime=1463533070990, allocationTime=1463533070990, startTime=1463533070992, finishTime=1463533071023, timeTaken=31, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:51,023 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533070744_0001_1_01_000000_0
2016-05-18 00:57:51,023 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:TASK_FINISHED]: vertexName=EC4B65F847404BD89201506981CE010A, taskId=task_1463533070744_0001_1_01_000000, startTime=1463533070992, finishTime=1463533071023, timeTaken=31, status=SUCCEEDED, successfulAttemptID=attempt_1463533070744_0001_1_01_000000_0, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=6, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533071015, Process_Duration=6, Process_End_Time=1463533071021, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20
2016-05-18 00:57:51,024 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:51,024 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A]
2016-05-18 00:57:51,024 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533070744_0001_00_000002
2016-05-18 00:57:51,024 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:VERTEX_FINISHED]: vertexName=EC4B65F847404BD89201506981CE010A, vertexId=vertex_1463533070744_0001_1_01, initRequestedTime=1463533070912, initedTime=1463533070916, startRequestedTime=1463533070922, startedTime=1463533070922, finishTime=1463533071024, timeTaken=102, status=SUCCEEDED, diagnostics=, counters=Counters: 36, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=20, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=6, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533071015, Process_Duration=6, Process_End_Time=1463533071021, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=20, vertexStats=firstTaskStartTime=1463533070992, firstTasksToStart=[ task_1463533070744_0001_1_01_000000 ], lastTaskFinishTime=1463533071023, lastTasksToFinish=[ task_1463533070744_0001_1_01_000000 ], minTaskDuration=31, maxTaskDuration=31, avgTaskDuration=31.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533070744_0001_1_01_000000 ], longestDurationTasks=[ task_1463533070744_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:51,025 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:51,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533070744_0001_00_000002, stoppedTime=1463533071025, exitStatus=0
2016-05-18 00:57:51,025 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:51,025 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:51,025 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1051)) - No output committers for vertex: vertex_1463533070744_0001_1_00 [ABCFC83C9AF147DEA084F8D993F4BC9C]
2016-05-18 00:57:51,025 INFO impl.DAGImpl (DAGImpl.java:commitOrFinish(1098)) - Start writing dag commit event, dag_1463533070744_0001_1
2016-05-18 00:57:51,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463533070744_0001_1
2016-05-18 00:57:51,025 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533070744_0001_1 transitioned from RUNNING to COMMITTING due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:51,025 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533070744_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-18 00:57:51,026 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533070744_0001_00_000002. Relying on regular task shutdown for it to end
2016-05-18 00:57:51,026 INFO impl.DAGImpl (DAGImpl.java:call(1083)) - Committing output: DBB8296F1DA54ADCA65B08320097550E for vertex: vertex_1463533070744_0001_1_01 [EC4B65F847404BD89201506981CE010A], outputName: DBB8296F1DA54ADCA65B08320097550E
2016-05-18 00:57:51,026 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533070744_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:51,026 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533070744_0001_00_000002, Exiting
2016-05-18 00:57:51,026 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533070744_0001_00_000002
2016-05-18 00:57:51,026 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533070744_0001_00_000002
2016-05-18 00:57:51,026 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533070744_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:51,026 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533070744_0001_00_000002 completed successfully
2016-05-18 00:57:51,032 INFO impl.DAGImpl (DAGImpl.java:commitCompleted(2192)) - Commit succeeded for output:outputName:DBB8296F1DA54ADCA65B08320097550E of vertex/vertexGroup:EC4B65F847404BD89201506981CE010A isVertexGroupOutput:false
2016-05-18 00:57:51,033 INFO impl.DAGImpl (DAGImpl.java:checkCommitsForCompletion(1313)) - Checking commits for DAG completion, numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:51,034 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533070744_0001_1, queueSize=0
2016-05-18 00:57:51,034 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533070744_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533070744_0001_1, startTime=1463533070911, finishTime=1463533071033, timeTaken=122, status=SUCCEEDED, diagnostics=, counters=Counters: 48, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=230, AM_GC_TIME_MILLIS=12, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=20, REDUCE_INPUT_RECORDS=20, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=40, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=40, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=6, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927066141993, Process_Duration=8, Process_End_Time=2927066142001, Read_Duration=3, Tuples_Read=40, Tuples_Written=40, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, Tuples_Written=20
2016-05-18 00:57:51,034 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533070744_0001_1 finished with state: SUCCEEDED
2016-05-18 00:57:51,034 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533070744_0001_1 transitioned from COMMITTING to SUCCEEDED due to event DAG_COMMIT_COMPLETED
2016-05-18 00:57:51 Completed Dag: dag_1463533070744_0001_1
2016-05-18 00:57:51,035 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533070744_0001_1, dagState=SUCCEEDED
2016-05-18 00:57:51,035 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533070744_0001_1_post
2016-05-18 00:57:51,035 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:51,035 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:51,035 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:51,035 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C] every/(1/1) ...platform/ALL_UNKNOWN_SWAP, with id=dag_1463533070744_0001_1
2016-05-18 00:57:51,035 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C] every/(1/1) ...platform/ALL_UNKNOWN_SWAP, with id=dag_1463533070744_0001_1
2016-05-18 00:57:51,045 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...platform/ALL_UNKNOWN_SWAP, applicationId=application_1463533070744_0001
2016-05-18 00:57:51,046 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:51,047 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:51,047 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:51,047 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:51,047 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:51,047 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:51,048 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:51,048 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:51,048 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:51,048 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:51,049 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:51,049 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:51,049 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:51,050 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:51,053 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/FBA5C4BBAA1B45FCB327DD67E497E273/DE0B972498DF4E5D91FDBEE806C4A84C/.tez/application_1463533070744_0001
2016-05-18 00:57:51,054 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_UNKNOWN_SWAP/_temporary
2016-05-18 00:57:51,054 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.325
2016-05-18 00:57:51,135 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_UNKNOWN_SWAP
2016-05-18 00:57:51,167 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:51,168 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:51,220 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.008
2016-05-18 00:57:51,230 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.018
2016-05-18 00:57:51,231 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:51,231 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:51,231 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:51,252 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:51,259 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_-1
2016-05-18 00:57:51,324 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:51,324 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:51,324 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_-1"]
2016-05-18 00:57:51,324 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:51,324 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:51,324 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:51,325 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...blyplatform/ALL_VALUES_-1
2016-05-18 00:57:51,325 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:51,337 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:51,342 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:51,342 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:51,342 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:51,351 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001 doesn't exist and is created
2016-05-18 00:57:51,365 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:51,365 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:51,365 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001_wd
2016-05-18 00:57:51,375 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533071342_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:51,375 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:51,375 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:51,377 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:51,377 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:51,377 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:51,377 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:51,378 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:51,380 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:51,380 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:51,380 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:51,380 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:51,380 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:51,381 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:51,381 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:51,381 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:51,381 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:51,381 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:51,381 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:51,381 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:51,383 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:51,383 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:51,383 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:51,383 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001_wd/localmode-log-dir
2016-05-18 00:57:51,384 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001_wd/localmode-log-dir/history.txt.appattempt_1463533071342_0001_000000, maxErrors=10
2016-05-18 00:57:51,384 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:51,384 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533071342_0001_000000, appSubmitTime=1463533071374, launchTime=1463533071375
2016-05-18 00:57:51,387 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:51,389 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:51,389 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:51,390 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:59292
2016-05-18 00:57:51,390 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533071342
2016-05-18 00:57:51,391 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533071342_0001_000000, startTime=1463533071391
2016-05-18 00:57:51,391 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:51,465 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:51,466 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:51,466 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...blyplatform/ALL_VALUES_-1, applicationId=application_1463533071342_0001, dagName=[4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD] every/(1/1) ...blyplatform/ALL_VALUES_-1
2016-05-18 00:57:51,480 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD] every/(1/1) ...blyplatform/ALL_VALUES_-1
2016-05-18 00:57:51,481 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:51,481 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533071342_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001_wd/localmode-log-dir/dag_1463533071342_0001_1.dot
2016-05-18 00:57:51,482 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001_wd/localmode-log-dir/dag_1463533071342_0001_1-tez-dag.pb.txt
2016-05-18 00:57:51,488 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533071342_0001_1
2016-05-18 00:57:51,488 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:51,488 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD] every/(1/1) ...blyplatform/ALL_VALUES_-1
2016-05-18 00:57:51 Running Dag: dag_1463533071342_0001_1
2016-05-18 00:57:51,496 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533071342_0001_1, submitTime=1463533071480
2016-05-18 00:57:51,497 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001/tez-conf.pb" } size: 36913 timestamp: 1463533071000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:51,497 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,498 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:51,498 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001/tez-conf.pb" } size: 36913 timestamp: 1463533071000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:51,498 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,499 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:51,499 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:51,499 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533071342_0001_1, initTime=1463533071499
2016-05-18 00:57:51,499 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533071342_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:51,499 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:51,500 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:DAG_STARTED]: dagID=dag_1463533071342_0001_1, startTime=1463533071500
2016-05-18 00:57:51,500 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533071342_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:51,500 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: ACA72580795445EDA72932A9CD1E6176 : {EFD2C4FF96D4401E94075B128B6982BF={InputName=EFD2C4FF96D4401E94075B128B6982BF}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:51,500 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: EFD2C4FF96D4401E94075B128B6982BF, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:51,500 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,500 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,500 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,500 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,500 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:51,501 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: EFD2C4FF96D4401E94075B128B6982BF on vertex vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,501 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,502 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...blyplatform/ALL_VALUES_-1, applicationId=application_1463533071342_0001, dagName=[4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD] every/(1/1) ...blyplatform/ALL_VALUES_-1
2016-05-18 00:57:51,504 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:51,504 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,504 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,505 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533071342_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:51,505 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08], numAdditionalOutputs=1
2016-05-18 00:57:51,506 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=C1F05649E60E48C28CB7EBD5084CDC57, vertex=vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:51,506 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input EFD2C4FF96D4401E94075B128B6982BF asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:51,507 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:51,507 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:51,508 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:51,508 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:51,508 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for AA304878E85944FEBED3EAA6E0A16E08:C1F05649E60E48C28CB7EBD5084CDC57 using oldmapred API
2016-05-18 00:57:51,508 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=C1F05649E60E48C28CB7EBD5084CDC57, vertexName=AA304878E85944FEBED3EAA6E0A16E08, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:51,508 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: EFD2C4FF96D4401E94075B128B6982BF on vertex vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,510 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AA304878E85944FEBED3EAA6E0A16E08, vertexId=vertex_1463533071342_0001_1_01, initRequestedTime=1463533071501, initedTime=1463533071505, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:51,510 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:51,510 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:51,511 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {EFD2C4FF96D4401E94075B128B6982BF=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:51,511 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] parallelism set to 1
2016-05-18 00:57:51,511 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:51,512 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:VERTEX_INITIALIZED]: vertexName=ACA72580795445EDA72932A9CD1E6176, vertexId=vertex_1463533071342_0001_1_00, initRequestedTime=1463533071500, initedTime=1463533071512, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:51,512 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] with distanceFromRoot: 0
2016-05-18 00:57:51,513 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:51,513 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533071342_0001_1_00, startRequestedTime=1463533071501, startedTime=1463533071513
2016-05-18 00:57:51,514 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533071342_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:51,514 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:51,514 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533071342_0001_1_00 for vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] numStartedSources: 1 numSources: 1
2016-05-18 00:57:51,514 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in ACA72580795445EDA72932A9CD1E6176
2016-05-18 00:57:51,514 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,514 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] with distanceFromRoot: 1
2016-05-18 00:57:51,515 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533071342_0001_1_01, startRequestedTime=1463533071514, startedTime=1463533071515
2016-05-18 00:57:51,515 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:51,516 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: AA304878E85944FEBED3EAA6E0A16E08 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:51,516 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:TASK_STARTED]: vertexName=ACA72580795445EDA72932A9CD1E6176, taskId=task_1463533071342_0001_1_00_000000, scheduledTime=1463533071516, launchTime=1463533071516
2016-05-18 00:57:51,516 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: ACA72580795445EDA72932A9CD1E6176 in vertex: AA304878E85944FEBED3EAA6E0A16E08 numBipartiteSourceTasks: 1
2016-05-18 00:57:51,517 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,518 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:51,520 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533071342_0001_00_000001, launchTime=1463533071519
2016-05-18 00:57:51,520 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533071342_0001_00_000001
2016-05-18 00:57:51,520 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533071342_0001, with dag id: dag_1463533071342_0001_1
2016-05-18 00:57:51,521 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533071342_0001_1_00_000000_0] started. Is using containerId: [container_1463533071342_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:51,521 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=ACA72580795445EDA72932A9CD1E6176, taskAttemptId=attempt_1463533071342_0001_1_00_000000_0, startTime=1463533071520, containerId=container_1463533071342_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:51,522 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533071342_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533071342_0001_1_00_000000_0
2016-05-18 00:57:51 Starting to run new task attempt: attempt_1463533071342_0001_1_00_000000_0
2016-05-18 00:57:51,522 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533071342_0001_1_00_000000_0
2016-05-18 00:57:51,522 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:51,522 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:51,523 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD] every/(1/1) ...blyplatform/ALL_VALUES_-1, VertexName: ACA72580795445EDA72932A9CD1E6176, VertexParallelism: 1, TaskAttemptID:attempt_1463533071342_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=EFD2C4FF96D4401E94075B128B6982BF, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=AA304878E85944FEBED3EAA6E0A16E08, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:51,523 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:51,523 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533071342_0001_1_00_000000_0
2016-05-18 00:57:51,537 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533071342_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:51,538 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533071342_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:51,538 INFO input.MRInput (MRInput.java:initialize(440)) - EFD2C4FF96D4401E94075B128B6982BF using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:51,538 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: EFD2C4FF96D4401E94075B128B6982BF
2016-05-18 00:57:51,538 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:51,542 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: ACA72580795445EDA72932A9CD1E6176, ordinal: 0
2016-05-18 00:57:51,542 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ACA72580795445EDA72932A9CD1E6176, mem on start (mb), free: 269, total: 367, max: 672
2016-05-18 00:57:51,542 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:51,542 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:51,542 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:51,542 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:51,542 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[EFD2C4FF96D4401E94075B128B6982BF:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [AA304878E85944FEBED3EAA6E0A16E08:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:51,542 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:51,543 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:51,543 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:51,543 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[EFD2C4FF96D4401E94075B128B6982BF:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [AA304878E85944FEBED3EAA6E0A16E08:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:51,543 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:51,543 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: EFD2C4FF96D4401E94075B128B6982BF being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:51,543 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:51,543 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:51,543 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: EFD2C4FF96D4401E94075B128B6982BF
2016-05-18 00:57:51,543 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:51,544 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533071342_0001_1_00_000000_0
2016-05-18 00:57:51,545 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:51,545 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - EFD2C4FF96D4401E94075B128B6982BF initialized RecordReader from event
2016-05-18 00:57:51,546 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:51,546 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: EFD2C4FF96D4401E94075B128B6982BF
2016-05-18 00:57:51,546 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: C65E508B2C784E8ABC903D456F8871E8
2016-05-18 00:57:51,546 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] C65E508B2C784E8ABC903D456F8871E8
2016-05-18 00:57:51,547 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:51,549 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:51,549 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - AA304878E85944FEBED3EAA6E0A16E08 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@5a25694a, comparator=cascading.tuple.hadoop.util.TupleComparator@4e23a782, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:51,551 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for AA304878E85944FEBED3EAA6E0A16E08: , UsingHashComparator=false
2016-05-18 00:57:51,563 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:51,564 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:51,564 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for AA304878E85944FEBED3EAA6E0A16E08: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:51,564 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - AA304878E85944FEBED3EAA6E0A16E08: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:51,564 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:51,565 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: ACA72580795445EDA72932A9CD1E6176, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:51,566 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533071342_0001_1_00_000000_0
2016-05-18 00:57:51,567 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: ACA72580795445EDA72932A9CD1E6176, mem on close (mb), free: 167, total: 367, max: 672
2016-05-18 00:57:51,567 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - AA304878E85944FEBED3EAA6E0A16E08: Starting flush of map output
2016-05-18 00:57:51,567 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - AA304878E85944FEBED3EAA6E0A16E08: Span0.length = 20, perItem = 6
2016-05-18 00:57:51,567 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - AA304878E85944FEBED3EAA6E0A16E08: done sorting span=0, length=20, time=0
2016-05-18 00:57:51,567 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - AA304878E85944FEBED3EAA6E0A16E08: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:51,568 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - AA304878E85944FEBED3EAA6E0A16E08: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001_wd/localmode-local-dir/output/attempt_1463533071342_0001_1_00_000000_0_10291_0/file.out
2016-05-18 00:57:51,571 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:51 Completed running task attempt: attempt_1463533071342_0001_1_00_000000_0
2016-05-18 00:57:51,572 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533071342_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:51,572 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533071342_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:51,573 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533071342_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533071565, Process_Duration=1, Process_End_Time=1463533071566, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:51,573 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533071342_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:51,573 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:51,573 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:51,574 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:51,573 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=ACA72580795445EDA72932A9CD1E6176, taskAttemptId=attempt_1463533071342_0001_1_00_000000_0, creationTime=1463533071517, allocationTime=1463533071519, startTime=1463533071520, finishTime=1463533071573, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:51,574 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533071342_0001_00_000001
2016-05-18 00:57:51,574 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533071342_0001_1_00_000000_0
2016-05-18 00:57:51,574 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:TASK_FINISHED]: vertexName=ACA72580795445EDA72932A9CD1E6176, taskId=task_1463533071342_0001_1_00_000000, startTime=1463533071520, finishTime=1463533071574, timeTaken=54, status=SUCCEEDED, successfulAttemptID=attempt_1463533071342_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533071565, Process_Duration=1, Process_End_Time=1463533071566, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:51,575 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:51,575 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176]
2016-05-18 00:57:51,575 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:VERTEX_FINISHED]: vertexName=ACA72580795445EDA72932A9CD1E6176, vertexId=vertex_1463533071342_0001_1_00, initRequestedTime=1463533071500, initedTime=1463533071512, startRequestedTime=1463533071501, startedTime=1463533071513, finishTime=1463533071575, timeTaken=62, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533071565, Process_Duration=1, Process_End_Time=1463533071566, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533071520, firstTasksToStart=[ task_1463533071342_0001_1_00_000000 ], lastTaskFinishTime=1463533071573, lastTasksToFinish=[ task_1463533071342_0001_1_00_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533071342_0001_1_00_000000 ], longestDurationTasks=[ task_1463533071342_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:51,575 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:51,576 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533071342_0001_00_000001, stoppedTime=1463533071575, exitStatus=0
2016-05-18 00:57:51,576 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] attempt: attempt_1463533071342_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:51,576 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533071342_0001_1_00 [ACA72580795445EDA72932A9CD1E6176] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:51,576 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:51,576 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533071342_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:51,576 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AA304878E85944FEBED3EAA6E0A16E08
2016-05-18 00:57:51,576 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=ACA72580795445EDA72932A9CD1E6176, destinationVertexName=AA304878E85944FEBED3EAA6E0A16E08. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:51,577 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,577 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:TASK_STARTED]: vertexName=AA304878E85944FEBED3EAA6E0A16E08, taskId=task_1463533071342_0001_1_01_000000, scheduledTime=1463533071577, launchTime=1463533071577
2016-05-18 00:57:51,577 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,579 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533071342_0001_00_000002, launchTime=1463533071579
2016-05-18 00:57:51,580 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533071342_0001_00_000002
2016-05-18 00:57:51,580 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533071342_0001_1_01_000000_0] started. Is using containerId: [container_1463533071342_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:51,580 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AA304878E85944FEBED3EAA6E0A16E08, taskAttemptId=attempt_1463533071342_0001_1_01_000000_0, startTime=1463533071580, containerId=container_1463533071342_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:51,581 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533071342_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533071342_0001_1_01_000000_0
2016-05-18 00:57:51 Starting to run new task attempt: attempt_1463533071342_0001_1_01_000000_0
2016-05-18 00:57:51,581 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533071342_0001_1_01_000000_0
2016-05-18 00:57:51,581 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:51,581 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:51,582 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD] every/(1/1) ...blyplatform/ALL_VALUES_-1, VertexName: AA304878E85944FEBED3EAA6E0A16E08, VertexParallelism: 1, TaskAttemptID:attempt_1463533071342_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ACA72580795445EDA72932A9CD1E6176, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=C1F05649E60E48C28CB7EBD5084CDC57, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:51,582 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:51,582 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533071342_0001_1_01_000000_0
2016-05-18 00:57:51,584 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533071342_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:51,584 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533071342_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:51,584 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533071342_0001_00_000001, Exiting
2016-05-18 00:57:51,584 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533071342_0001_00_000001
2016-05-18 00:57:51,584 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533071342_0001_00_000001
2016-05-18 00:57:51,585 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:51,586 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:51,586 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533071342_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,586 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533071342_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:51,586 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533071342_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:51,587 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533071342_0001_00_000001 completed successfully
2016-05-18 00:57:51,589 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AA304878E85944FEBED3EAA6E0A16E08, ordinal: 1
2016-05-18 00:57:51,589 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AA304878E85944FEBED3EAA6E0A16E08, mem on start (mb), free: 162, total: 367, max: 672
2016-05-18 00:57:51,589 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:51,589 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:51,589 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:51,594 INFO output.MROutput (MROutput.java:initialize(421)) - C1F05649E60E48C28CB7EBD5084CDC57: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:51,594 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:51,594 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[ACA72580795445EDA72932A9CD1E6176:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [C1F05649E60E48C28CB7EBD5084CDC57:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:51,594 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:51,594 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:51,595 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:51,595 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[ACA72580795445EDA72932A9CD1E6176:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [C1F05649E60E48C28CB7EBD5084CDC57:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:51,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:51,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: ACA72580795445EDA72932A9CD1E6176 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:51,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:51,595 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:51,595 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - ACA72580795445EDA72932A9CD1E6176: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:51,595 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - ACA72580795445EDA72932A9CD1E6176: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:51,597 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: ACA72580795445EDA72932A9CD1E6176 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:51,597 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: ACA72580795445EDA72932A9CD1E6176
2016-05-18 00:57:51,597 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:51,597 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533071342_0001_1_01_000000_0
2016-05-18 00:57:51,598 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - ACA72580795445EDA72932A9CD1E6176: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:51,599 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533071342_0001_1_00_000000_0_10291} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533071599, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:51,599 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : ACA72580795445EDA72932A9CD1E6176
2016-05-18 00:57:51,600 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:51,600 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: ACA72580795445EDA72932A9CD1E6176, wasInterrupted=false
2016-05-18 00:57:51,600 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:51,600 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:51,601 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: C65E508B2C784E8ABC903D456F8871E8
2016-05-18 00:57:51,601 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:51,601 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_-1"], id: C1F05649E60E48C28CB7EBD5084CDC57
2016-05-18 00:57:51,602 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_-1"]
2016-05-18 00:57:51,602 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:51,602 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001_wd/localmode-local-dir/output/attempt_1463533071342_0001_1_00_000000_0_10291/file.out, len=181, isLocal=true
2016-05-18 00:57:51,602 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] C65E508B2C784E8ABC903D456F8871E8, for 1 inputs
2016-05-18 00:57:51,603 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:51,603 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:51,603 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:51,603 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:51,603 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : ACA72580795445EDA72932A9CD1E6176
2016-05-18 00:57:51,603 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - ACA72580795445EDA72932A9CD1E6176: Shuffle Runner thread complete
2016-05-18 00:57:51,603 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AA304878E85944FEBED3EAA6E0A16E08, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:51,604 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:51,604 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - ACA72580795445EDA72932A9CD1E6176: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:51,606 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:51,607 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:51,610 INFO output.MROutput (MROutput.java:close(526)) - C1F05649E60E48C28CB7EBD5084CDC57 closed
2016-05-18 00:57:51,611 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533071342_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:51,611 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533071342_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:51,612 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533071342_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533071604, Process_Duration=7, Process_End_Time=1463533071611, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:51,612 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:51,612 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AA304878E85944FEBED3EAA6E0A16E08, mem on close (mb), free: 159, total: 367, max: 672
2016-05-18 00:57:51,612 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=AA304878E85944FEBED3EAA6E0A16E08, index=1, interruptedStatus=false
2016-05-18 00:57:51,612 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: ACA72580795445EDA72932A9CD1E6176
2016-05-18 00:57:51,613 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - ACA72580795445EDA72932A9CD1E6176: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:51,614 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=AA304878E85944FEBED3EAA6E0A16E08, sourceVertex=ACA72580795445EDA72932A9CD1E6176, interruptedStatus=true
2016-05-18 00:57:51,614 INFO output.MROutput (MROutput.java:close(526)) - C1F05649E60E48C28CB7EBD5084CDC57 closed
2016-05-18 00:57:51,614 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=AA304878E85944FEBED3EAA6E0A16E08, sourceVertex=C1F05649E60E48C28CB7EBD5084CDC57, interruptedStatus=true
2016-05-18 00:57:51,615 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533071342_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:51,615 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:51,615 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533071342_0001_00_000002
2016-05-18 00:57:51,615 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533071342_0001_00_000002
2016-05-18 00:57:51,615 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AA304878E85944FEBED3EAA6E0A16E08, taskAttemptId=attempt_1463533071342_0001_1_01_000000_0, creationTime=1463533071577, allocationTime=1463533071579, startTime=1463533071580, finishTime=1463533071615, timeTaken=35, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533071342_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=8, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533071604, Process_Duration=7, Process_End_Time=1463533071611, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:51,615 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533071342_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:51,615 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533071342_0001_00_000002 completed but with errors
2016-05-18 00:57:51,615 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533071342_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:51,616 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:TASK_FINISHED]: vertexName=AA304878E85944FEBED3EAA6E0A16E08, taskId=task_1463533071342_0001_1_01_000000, startTime=1463533071580, finishTime=1463533071616, timeTaken=36, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533071342_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:51,616 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533071342_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:51,616 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533071342_0001_00_000002, stoppedTime=1463533071616, exitStatus=1
2016-05-18 00:57:51,616 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] because task failed: task_1463533071342_0001_1_01_000000
2016-05-18 00:57:51,616 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:51,616 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:51,616 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:51,617 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,617 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=C1F05649E60E48C28CB7EBD5084CDC57, vertexId=vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,618 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:VERTEX_FINISHED]: vertexName=AA304878E85944FEBED3EAA6E0A16E08, vertexId=vertex_1463533071342_0001_1_01, initRequestedTime=1463533071501, initedTime=1463533071505, startRequestedTime=1463533071514, startedTime=1463533071515, finishTime=1463533071618, timeTaken=103, status=FAILED, diagnostics=Task failed, taskId=task_1463533071342_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533071342_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533071580, firstTasksToStart=[ task_1463533071342_0001_1_01_000000 ], lastTaskFinishTime=1463533071615, lastTasksToFinish=[ task_1463533071342_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:51,618 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:51,618 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533071342_0001_1_01_000000_0 failed: true container: container_1463533071342_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:51,618 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:51,619 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:51,619 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:51,620 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533071342_0001_1, queueSize=0
2016-05-18 00:57:51,620 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071342_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533071342_0001_1, startTime=1463533071499, finishTime=1463533071619, timeTaken=120, status=FAILED, diagnostics=Vertex failed, vertexName=AA304878E85944FEBED3EAA6E0A16E08, vertexId=vertex_1463533071342_0001_1_01, diagnostics=[Task failed, taskId=task_1463533071342_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533071342_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=200, AM_GC_TIME_MILLIS=10, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533071565, Process_Duration=1, Process_End_Time=1463533071566, Read_Duration=0, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:51,621 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533071342_0001_1_01 [AA304878E85944FEBED3EAA6E0A16E08]
2016-05-18 00:57:51,622 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533071342_0001_1 finished with state: FAILED
2016-05-18 00:57:51,622 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533071342_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:51 Completed Dag: dag_1463533071342_0001_1
2016-05-18 00:57:51,622 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533071342_0001_1, dagState=FAILED
2016-05-18 00:57:51,622 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533071342_0001_1_post
2016-05-18 00:57:51,623 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:51,623 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:51,623 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:51,623 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD] every/(1/1) ...blyplatform/ALL_VALUES_-1, with id=dag_1463533071342_0001_1
2016-05-18 00:57:51,623 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD] every/(1/1) ...blyplatform/ALL_VALUES_-1, with id=dag_1463533071342_0001_1
2016-05-18 00:57:51,631 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...blyplatform/ALL_VALUES_-1, applicationId=application_1463533071342_0001
2016-05-18 00:57:51,633 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:51,633 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:51,633 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:51,633 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:51,633 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:51,633 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:51,634 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:51,634 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:51,634 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:51,634 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:51,634 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:51,634 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:51,635 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:51,635 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:51,639 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/4602D537E8D74FB2990B207F1EF1ED3D/EBE0BA0CB9404185937D2DD4A7B9E2AD/.tez/application_1463533071342_0001
2016-05-18 00:57:51,639 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:51,640 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...blyplatform/ALL_VALUES_-1
2016-05-18 00:57:51,640 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:51,640 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_-1/_temporary
2016-05-18 00:57:51,640 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.316
2016-05-18 00:57:51,640 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_VALUES_-1
2016-05-18 00:57:51,674 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:51,674 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:51,707 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.002
2016-05-18 00:57:51,709 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.004
2016-05-18 00:57:51,709 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:51,709 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:51,710 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_VALUES_'field'
2016-05-18 00:57:51,732 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:51,733 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:51,762 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.005
2016-05-18 00:57:51,769 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 00:57:51,770 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:51,770 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:51,770 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:51,785 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:51,790 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_RESULTS
2016-05-18 00:57:51,838 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:51,838 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:51,838 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_RESULTS"]
2016-05-18 00:57:51,838 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:51,839 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:51,839 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:51,839 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...atform/ALL_VALUES_RESULTS
2016-05-18 00:57:51,839 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:51,853 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:51,858 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:51,858 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:51,858 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:51,867 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001 doesn't exist and is created
2016-05-18 00:57:51,881 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:51,881 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:51,882 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001_wd
2016-05-18 00:57:51,891 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533071858_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:51,891 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:51,891 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:51,892 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:51,893 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:51,893 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:51,893 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:51,893 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:51,895 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:51,895 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:51,895 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:51,895 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:51,895 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:51,895 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:51,895 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:51,895 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:51,895 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:51,896 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:51,896 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:51,896 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:51,897 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:51,898 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:51,898 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:51,899 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001_wd/localmode-log-dir
2016-05-18 00:57:51,899 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001_wd/localmode-log-dir/history.txt.appattempt_1463533071858_0001_000000, maxErrors=10
2016-05-18 00:57:51,899 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:51,899 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533071858_0001_000000, appSubmitTime=1463533071890, launchTime=1463533071891
2016-05-18 00:57:51,902 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:51,905 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:52721
2016-05-18 00:57:51,905 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533071858
2016-05-18 00:57:51,906 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533071858_0001_000000, startTime=1463533071906
2016-05-18 00:57:51,907 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:51,914 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:51,914 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:51,982 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:51,982 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:51,982 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...atform/ALL_VALUES_RESULTS, applicationId=application_1463533071858_0001, dagName=[E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5] every/(1/1) ...atform/ALL_VALUES_RESULTS
2016-05-18 00:57:51,996 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5] every/(1/1) ...atform/ALL_VALUES_RESULTS
2016-05-18 00:57:51,997 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:51,997 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533071858_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001_wd/localmode-log-dir/dag_1463533071858_0001_1.dot
2016-05-18 00:57:51,998 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001_wd/localmode-log-dir/dag_1463533071858_0001_1-tez-dag.pb.txt
2016-05-18 00:57:52,004 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533071858_0001_1
2016-05-18 00:57:52,004 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,004 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5] every/(1/1) ...atform/ALL_VALUES_RESULTS
2016-05-18 00:57:52 Running Dag: dag_1463533071858_0001_1
2016-05-18 00:57:52,012 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533071858_0001_1, submitTime=1463533071996
2016-05-18 00:57:52,013 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001/tez-conf.pb" } size: 36913 timestamp: 1463533071000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:52,013 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,014 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:52,015 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001/tez-conf.pb" } size: 36913 timestamp: 1463533071000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:52,015 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,015 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:52,016 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:52,017 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533071858_0001_1, initTime=1463533072016
2016-05-18 00:57:52,017 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533071858_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:52,017 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:52,017 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:DAG_STARTED]: dagID=dag_1463533071858_0001_1, startTime=1463533072017
2016-05-18 00:57:52,018 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533071858_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:52,018 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: F02CA8FB6DCB4C8491FC6B6F50D919A3 : {53E5C46E9B7C44C9B59ED26634D47295={InputName=53E5C46E9B7C44C9B59ED26634D47295}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:52,018 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 53E5C46E9B7C44C9B59ED26634D47295, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:52,018 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,018 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,018 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,018 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,019 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:52,019 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 53E5C46E9B7C44C9B59ED26634D47295 on vertex vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,019 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,020 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...atform/ALL_VALUES_RESULTS, applicationId=application_1463533071858_0001, dagName=[E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5] every/(1/1) ...atform/ALL_VALUES_RESULTS
2016-05-18 00:57:52,020 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 53E5C46E9B7C44C9B59ED26634D47295 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:52,021 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:52,022 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:52,022 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:52,022 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:52,022 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,022 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,022 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533071858_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:52,023 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161], numAdditionalOutputs=1
2016-05-18 00:57:52,023 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=50347B5E5E3E48D5AED48D172A5E95C4, vertex=vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:52,022 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:52,024 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 53E5C46E9B7C44C9B59ED26634D47295 on vertex vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,024 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for D6D973C3CB324EDB97CF62D11C605161:50347B5E5E3E48D5AED48D172A5E95C4 using oldmapred API
2016-05-18 00:57:52,024 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=50347B5E5E3E48D5AED48D172A5E95C4, vertexName=D6D973C3CB324EDB97CF62D11C605161, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:52,025 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D6D973C3CB324EDB97CF62D11C605161, vertexId=vertex_1463533071858_0001_1_01, initRequestedTime=1463533072019, initedTime=1463533072023, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:52,025 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:52,026 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:52,026 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {53E5C46E9B7C44C9B59ED26634D47295=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:52,026 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] parallelism set to 1
2016-05-18 00:57:52,026 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:52,026 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F02CA8FB6DCB4C8491FC6B6F50D919A3, vertexId=vertex_1463533071858_0001_1_00, initRequestedTime=1463533072018, initedTime=1463533072026, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:52,026 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] with distanceFromRoot: 0
2016-05-18 00:57:52,026 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:52,027 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533071858_0001_1_00, startRequestedTime=1463533072019, startedTime=1463533072027
2016-05-18 00:57:52,027 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533071858_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:52,027 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:52,027 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533071858_0001_1_00 for vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] numStartedSources: 1 numSources: 1
2016-05-18 00:57:52,027 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] with distanceFromRoot: 1
2016-05-18 00:57:52,028 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533071858_0001_1_01, startRequestedTime=1463533072027, startedTime=1463533072027
2016-05-18 00:57:52,028 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:52,029 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in F02CA8FB6DCB4C8491FC6B6F50D919A3
2016-05-18 00:57:52,029 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: D6D973C3CB324EDB97CF62D11C605161 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:52,029 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,029 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:TASK_STARTED]: vertexName=F02CA8FB6DCB4C8491FC6B6F50D919A3, taskId=task_1463533071858_0001_1_00_000000, scheduledTime=1463533072029, launchTime=1463533072029
2016-05-18 00:57:52,029 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: F02CA8FB6DCB4C8491FC6B6F50D919A3 in vertex: D6D973C3CB324EDB97CF62D11C605161 numBipartiteSourceTasks: 1
2016-05-18 00:57:52,030 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,030 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:52,032 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533071858_0001_00_000001, launchTime=1463533072031
2016-05-18 00:57:52,032 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533071858_0001_00_000001
2016-05-18 00:57:52,032 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533071858_0001_1_00_000000_0] started. Is using containerId: [container_1463533071858_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:52,033 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F02CA8FB6DCB4C8491FC6B6F50D919A3, taskAttemptId=attempt_1463533071858_0001_1_00_000000_0, startTime=1463533072032, containerId=container_1463533071858_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:52,033 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533071858_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533071858_0001_1_00_000000_0
2016-05-18 00:57:52 Starting to run new task attempt: attempt_1463533071858_0001_1_00_000000_0
2016-05-18 00:57:52,034 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533071858_0001_1_00_000000_0
2016-05-18 00:57:52,034 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,034 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:52,035 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5] every/(1/1) ...atform/ALL_VALUES_RESULTS, VertexName: F02CA8FB6DCB4C8491FC6B6F50D919A3, VertexParallelism: 1, TaskAttemptID:attempt_1463533071858_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=53E5C46E9B7C44C9B59ED26634D47295, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=D6D973C3CB324EDB97CF62D11C605161, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:52,035 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:52,035 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533071858_0001_1_00_000000_0
2016-05-18 00:57:52,036 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533071858_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:52,036 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533071858_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:52,039 INFO input.MRInput (MRInput.java:initialize(440)) - 53E5C46E9B7C44C9B59ED26634D47295 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:52,039 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 53E5C46E9B7C44C9B59ED26634D47295
2016-05-18 00:57:52,039 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:52,043 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F02CA8FB6DCB4C8491FC6B6F50D919A3, ordinal: 0
2016-05-18 00:57:52,043 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F02CA8FB6DCB4C8491FC6B6F50D919A3, mem on start (mb), free: 55, total: 367, max: 672
2016-05-18 00:57:52,043 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:52,043 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:52,043 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:52,043 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:52,043 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[53E5C46E9B7C44C9B59ED26634D47295:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [D6D973C3CB324EDB97CF62D11C605161:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:52,043 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:52,043 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:52,043 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:52,044 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[53E5C46E9B7C44C9B59ED26634D47295:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [D6D973C3CB324EDB97CF62D11C605161:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:52,044 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:52,044 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 53E5C46E9B7C44C9B59ED26634D47295 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:52,044 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:52,044 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:52,044 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 53E5C46E9B7C44C9B59ED26634D47295
2016-05-18 00:57:52,045 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533071858_0001, with dag id: dag_1463533071858_0001_1
2016-05-18 00:57:52,045 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:52,045 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533071858_0001_1_00_000000_0
2016-05-18 00:57:52,059 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:52,060 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 53E5C46E9B7C44C9B59ED26634D47295 initialized RecordReader from event
2016-05-18 00:57:52,060 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:52,060 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 53E5C46E9B7C44C9B59ED26634D47295
2016-05-18 00:57:52,060 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: A7E7D68387674DF49CDB1CA7E52E4835
2016-05-18 00:57:52,060 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] A7E7D68387674DF49CDB1CA7E52E4835
2016-05-18 00:57:52,062 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,064 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,064 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - D6D973C3CB324EDB97CF62D11C605161 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@325d8b7f, comparator=cascading.tuple.hadoop.util.TupleComparator@1b7158ae, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:52,064 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for D6D973C3CB324EDB97CF62D11C605161: , UsingHashComparator=false
2016-05-18 00:57:52,077 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:52,077 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:52,077 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for D6D973C3CB324EDB97CF62D11C605161: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:52,077 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - D6D973C3CB324EDB97CF62D11C605161: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:52,078 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:52,080 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F02CA8FB6DCB4C8491FC6B6F50D919A3, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:57:52,081 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533071858_0001_1_00_000000_0
2016-05-18 00:57:52,082 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F02CA8FB6DCB4C8491FC6B6F50D919A3, mem on close (mb), free: 169, total: 368, max: 672
2016-05-18 00:57:52,082 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - D6D973C3CB324EDB97CF62D11C605161: Starting flush of map output
2016-05-18 00:57:52,082 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - D6D973C3CB324EDB97CF62D11C605161: Span0.length = 20, perItem = 6
2016-05-18 00:57:52,083 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - D6D973C3CB324EDB97CF62D11C605161: done sorting span=0, length=20, time=1
2016-05-18 00:57:52,083 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - D6D973C3CB324EDB97CF62D11C605161: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:52,084 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - D6D973C3CB324EDB97CF62D11C605161: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001_wd/localmode-local-dir/output/attempt_1463533071858_0001_1_00_000000_0_10297_0/file.out
2016-05-18 00:57:52,090 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:52 Completed running task attempt: attempt_1463533071858_0001_1_00_000000_0
2016-05-18 00:57:52,090 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533071858_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:52,091 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533071858_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:52,091 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533071858_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533072080, Process_Duration=1, Process_End_Time=1463533072081, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:52,091 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533071858_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:52,091 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:52,091 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F02CA8FB6DCB4C8491FC6B6F50D919A3, taskAttemptId=attempt_1463533071858_0001_1_00_000000_0, creationTime=1463533072029, allocationTime=1463533072031, startTime=1463533072032, finishTime=1463533072091, timeTaken=59, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:52,092 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:52,092 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,092 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533071858_0001_1_00_000000_0
2016-05-18 00:57:52,092 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:TASK_FINISHED]: vertexName=F02CA8FB6DCB4C8491FC6B6F50D919A3, taskId=task_1463533071858_0001_1_00_000000, startTime=1463533072032, finishTime=1463533072092, timeTaken=60, status=SUCCEEDED, successfulAttemptID=attempt_1463533071858_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072080, Process_Duration=1, Process_End_Time=1463533072081, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:52,092 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533071858_0001_00_000001
2016-05-18 00:57:52,092 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:52,093 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3]
2016-05-18 00:57:52,093 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:VERTEX_FINISHED]: vertexName=F02CA8FB6DCB4C8491FC6B6F50D919A3, vertexId=vertex_1463533071858_0001_1_00, initRequestedTime=1463533072018, initedTime=1463533072026, startRequestedTime=1463533072019, startedTime=1463533072027, finishTime=1463533072093, timeTaken=66, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072080, Process_Duration=1, Process_End_Time=1463533072081, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533072032, firstTasksToStart=[ task_1463533071858_0001_1_00_000000 ], lastTaskFinishTime=1463533072091, lastTasksToFinish=[ task_1463533071858_0001_1_00_000000 ], minTaskDuration=59, maxTaskDuration=59, avgTaskDuration=59.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533071858_0001_1_00_000000 ], longestDurationTasks=[ task_1463533071858_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:52,093 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:52,093 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] attempt: attempt_1463533071858_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:52,093 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: D6D973C3CB324EDB97CF62D11C605161
2016-05-18 00:57:52,094 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=F02CA8FB6DCB4C8491FC6B6F50D919A3, destinationVertexName=D6D973C3CB324EDB97CF62D11C605161. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:52,094 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,093 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533071858_0001_00_000001, stoppedTime=1463533072093, exitStatus=0
2016-05-18 00:57:52,094 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533071858_0001_1_00 [F02CA8FB6DCB4C8491FC6B6F50D919A3] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:52,094 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:52,094 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:TASK_STARTED]: vertexName=D6D973C3CB324EDB97CF62D11C605161, taskId=task_1463533071858_0001_1_01_000000, scheduledTime=1463533072094, launchTime=1463533072094
2016-05-18 00:57:52,095 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,095 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533071858_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:52,095 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533071858_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:52,096 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533071858_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:52,096 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533071858_0001_00_000001, Exiting
2016-05-18 00:57:52,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533071858_0001_00_000002, launchTime=1463533072096
2016-05-18 00:57:52,096 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533071858_0001_00_000001
2016-05-18 00:57:52,096 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533071858_0001_00_000001
2016-05-18 00:57:52,096 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533071858_0001_00_000002
2016-05-18 00:57:52,097 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533071858_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:52,097 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533071858_0001_00_000001 completed successfully
2016-05-18 00:57:52,097 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533071858_0001_1_01_000000_0] started. Is using containerId: [container_1463533071858_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:52,097 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D6D973C3CB324EDB97CF62D11C605161, taskAttemptId=attempt_1463533071858_0001_1_01_000000_0, startTime=1463533072097, containerId=container_1463533071858_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:52,097 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533071858_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533071858_0001_1_01_000000_0
2016-05-18 00:57:52 Starting to run new task attempt: attempt_1463533071858_0001_1_01_000000_0
2016-05-18 00:57:52,097 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533071858_0001_1_01_000000_0
2016-05-18 00:57:52,098 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,098 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:52,098 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5] every/(1/1) ...atform/ALL_VALUES_RESULTS, VertexName: D6D973C3CB324EDB97CF62D11C605161, VertexParallelism: 1, TaskAttemptID:attempt_1463533071858_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=F02CA8FB6DCB4C8491FC6B6F50D919A3, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=50347B5E5E3E48D5AED48D172A5E95C4, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:52,098 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:52,098 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533071858_0001_1_01_000000_0
2016-05-18 00:57:52,099 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533071858_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,100 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533071858_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:52,100 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:52,102 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:52,105 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D6D973C3CB324EDB97CF62D11C605161, ordinal: 1
2016-05-18 00:57:52,105 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D6D973C3CB324EDB97CF62D11C605161, mem on start (mb), free: 164, total: 368, max: 672
2016-05-18 00:57:52,105 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:52,105 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:52,105 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:52,110 INFO output.MROutput (MROutput.java:initialize(421)) - 50347B5E5E3E48D5AED48D172A5E95C4: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:52,110 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:52,110 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[F02CA8FB6DCB4C8491FC6B6F50D919A3:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [50347B5E5E3E48D5AED48D172A5E95C4:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:52,110 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:52,110 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:52,110 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:52,110 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[F02CA8FB6DCB4C8491FC6B6F50D919A3:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [50347B5E5E3E48D5AED48D172A5E95C4:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:52,110 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:52,111 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: F02CA8FB6DCB4C8491FC6B6F50D919A3 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:52,111 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:52,111 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:52,111 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - F02CA8FB6DCB4C8491FC6B6F50D919A3: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:52,111 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - F02CA8FB6DCB4C8491FC6B6F50D919A3: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:52,112 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: F02CA8FB6DCB4C8491FC6B6F50D919A3 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:52,113 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: F02CA8FB6DCB4C8491FC6B6F50D919A3
2016-05-18 00:57:52,114 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:52,114 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533071858_0001_1_01_000000_0
2016-05-18 00:57:52,116 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:52,117 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: A7E7D68387674DF49CDB1CA7E52E4835
2016-05-18 00:57:52,117 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_RESULTS"], id: 50347B5E5E3E48D5AED48D172A5E95C4
2016-05-18 00:57:52,117 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_RESULTS"]
2016-05-18 00:57:52,117 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] A7E7D68387674DF49CDB1CA7E52E4835, for 1 inputs
2016-05-18 00:57:52,117 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F02CA8FB6DCB4C8491FC6B6F50D919A3: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:52,118 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533071858_0001_1_00_000000_0_10297} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533072118, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:52,118 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : F02CA8FB6DCB4C8491FC6B6F50D919A3
2016-05-18 00:57:52,118 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:52,119 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: F02CA8FB6DCB4C8491FC6B6F50D919A3, wasInterrupted=false
2016-05-18 00:57:52,119 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:52,119 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - F02CA8FB6DCB4C8491FC6B6F50D919A3: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:52,119 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:52,120 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,120 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001_wd/localmode-local-dir/output/attempt_1463533071858_0001_1_00_000000_0_10297/file.out, len=181, isLocal=true
2016-05-18 00:57:52,120 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:52,120 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:52,120 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:52,121 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:52,121 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : F02CA8FB6DCB4C8491FC6B6F50D919A3
2016-05-18 00:57:52,121 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - F02CA8FB6DCB4C8491FC6B6F50D919A3: Shuffle Runner thread complete
2016-05-18 00:57:52,121 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D6D973C3CB324EDB97CF62D11C605161, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:52,122 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,122 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - F02CA8FB6DCB4C8491FC6B6F50D919A3: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:52,124 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:52,125 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:52,126 INFO output.MROutput (MROutput.java:close(526)) - 50347B5E5E3E48D5AED48D172A5E95C4 closed
2016-05-18 00:57:52,126 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533071858_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:52,127 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533071858_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:52,127 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533071858_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533072121, Process_Duration=6, Process_End_Time=1463533072127, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:52,127 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:52,128 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D6D973C3CB324EDB97CF62D11C605161, mem on close (mb), free: 160, total: 368, max: 672
2016-05-18 00:57:52,128 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=D6D973C3CB324EDB97CF62D11C605161, index=1, interruptedStatus=false
2016-05-18 00:57:52,128 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: F02CA8FB6DCB4C8491FC6B6F50D919A3
2016-05-18 00:57:52,128 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - F02CA8FB6DCB4C8491FC6B6F50D919A3: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:52,128 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=D6D973C3CB324EDB97CF62D11C605161, sourceVertex=F02CA8FB6DCB4C8491FC6B6F50D919A3, interruptedStatus=true
2016-05-18 00:57:52,128 INFO output.MROutput (MROutput.java:close(526)) - 50347B5E5E3E48D5AED48D172A5E95C4 closed
2016-05-18 00:57:52,129 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=D6D973C3CB324EDB97CF62D11C605161, sourceVertex=50347B5E5E3E48D5AED48D172A5E95C4, interruptedStatus=true
2016-05-18 00:57:52,129 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533071858_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:52,129 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:52,130 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533071858_0001_00_000002
2016-05-18 00:57:52,130 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D6D973C3CB324EDB97CF62D11C605161, taskAttemptId=attempt_1463533071858_0001_1_01_000000_0, creationTime=1463533072094, allocationTime=1463533072095, startTime=1463533072097, finishTime=1463533072129, timeTaken=32, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533071858_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=8, MERGE_PHASE_TIME=10, FIRST_EVENT_RECEIVED=6, LAST_EVENT_RECEIVED=6, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533072121, Process_Duration=6, Process_End_Time=1463533072127, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:52,130 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533071858_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:52,130 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533071858_0001_00_000002
2016-05-18 00:57:52,130 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:TASK_FINISHED]: vertexName=D6D973C3CB324EDB97CF62D11C605161, taskId=task_1463533071858_0001_1_01_000000, startTime=1463533072097, finishTime=1463533072130, timeTaken=33, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533071858_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:52,130 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533071858_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:52,130 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533071858_0001_00_000002 completed but with errors
2016-05-18 00:57:52,130 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] because task failed: task_1463533071858_0001_1_01_000000
2016-05-18 00:57:52,130 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:52,131 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:52,131 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:52,131 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,131 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=50347B5E5E3E48D5AED48D172A5E95C4, vertexId=vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,132 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:VERTEX_FINISHED]: vertexName=D6D973C3CB324EDB97CF62D11C605161, vertexId=vertex_1463533071858_0001_1_01, initRequestedTime=1463533072019, initedTime=1463533072023, startRequestedTime=1463533072027, startedTime=1463533072027, finishTime=1463533072132, timeTaken=105, status=FAILED, diagnostics=Task failed, taskId=task_1463533071858_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533071858_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533072097, firstTasksToStart=[ task_1463533071858_0001_1_01_000000 ], lastTaskFinishTime=1463533072129, lastTasksToFinish=[ task_1463533071858_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:52,132 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:52,132 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533071858_0001_00_000002, stoppedTime=1463533072132, exitStatus=-1000
2016-05-18 00:57:52,132 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533071858_0001_1_01_000000_0 failed: true container: container_1463533071858_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:52,132 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533071858_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:52,132 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:52,132 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:52,132 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:52,133 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533071858_0001_1, queueSize=0
2016-05-18 00:57:52,133 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533071858_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533071858_0001_1, startTime=1463533072017, finishTime=1463533072133, timeTaken=116, status=FAILED, diagnostics=Vertex failed, vertexName=D6D973C3CB324EDB97CF62D11C605161, vertexId=vertex_1463533071858_0001_1_01, diagnostics=[Task failed, taskId=task_1463533071858_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533071858_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=200, AM_GC_TIME_MILLIS=9, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072080, Process_Duration=1, Process_End_Time=1463533072081, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:52,133 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533071858_0001_1_01 [D6D973C3CB324EDB97CF62D11C605161]
2016-05-18 00:57:52,133 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533071858_0001_1 finished with state: FAILED
2016-05-18 00:57:52,133 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533071858_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:52,134 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533071858_0001_00_000002
2016-05-18 00:57:52 Completed Dag: dag_1463533071858_0001_1
2016-05-18 00:57:52,134 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533071858_0001_1, dagState=FAILED
2016-05-18 00:57:52,134 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533071858_0001_1_post
2016-05-18 00:57:52,134 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,134 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:52,134 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:52,135 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5] every/(1/1) ...atform/ALL_VALUES_RESULTS, with id=dag_1463533071858_0001_1
2016-05-18 00:57:52,135 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5] every/(1/1) ...atform/ALL_VALUES_RESULTS, with id=dag_1463533071858_0001_1
2016-05-18 00:57:52,141 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...atform/ALL_VALUES_RESULTS, applicationId=application_1463533071858_0001
2016-05-18 00:57:52,143 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:52,143 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:52,143 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:52,143 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:52,143 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:52,143 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:52,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:52,144 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:52,144 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:52,144 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:52,144 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:52,144 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:52,144 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:52,145 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:52,146 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E98DB123225349EBBF13958E1B0AC25A/AFB99C59B14F4AA7B4B0305DF0663EF5/.tez/application_1463533071858_0001
2016-05-18 00:57:52,146 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:52,147 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...atform/ALL_VALUES_RESULTS
2016-05-18 00:57:52,147 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:52,148 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_RESULTS/_temporary
2016-05-18 00:57:52,148 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.310
2016-05-18 00:57:52,148 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_VALUES_RESULTS
2016-05-18 00:57:52,172 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:52,172 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:52,202 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:52,209 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.012
2016-05-18 00:57:52,209 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:52,210 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:52,210 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:52,223 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:52,228 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_ALL
2016-05-18 00:57:52,272 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:52,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:52,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_ALL"]
2016-05-18 00:57:52,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:52,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:52,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:52,273 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...lyplatform/ALL_VALUES_ALL
2016-05-18 00:57:52,274 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:52,285 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:52,289 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:52,290 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:52,290 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:52,299 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001 doesn't exist and is created
2016-05-18 00:57:52,313 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:52,313 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:52,314 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001_wd
2016-05-18 00:57:52,323 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533072290_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:52,323 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:52,323 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:52,325 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:52,325 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:52,325 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:52,325 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:52,325 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:52,327 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:52,327 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:52,327 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:52,329 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:52,329 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:52,329 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:52,330 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001_wd/localmode-log-dir
2016-05-18 00:57:52,330 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001_wd/localmode-log-dir/history.txt.appattempt_1463533072290_0001_000000, maxErrors=10
2016-05-18 00:57:52,330 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:52,330 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533072290_0001_000000, appSubmitTime=1463533072322, launchTime=1463533072323
2016-05-18 00:57:52,333 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:52,336 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:43767
2016-05-18 00:57:52,336 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533072290
2016-05-18 00:57:52,338 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533072290_0001_000000, startTime=1463533072338
2016-05-18 00:57:52,338 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:52,413 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:52,414 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:52,414 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...lyplatform/ALL_VALUES_ALL, applicationId=application_1463533072290_0001, dagName=[5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864] every/(1/1) ...lyplatform/ALL_VALUES_ALL
2016-05-18 00:57:52,428 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864] every/(1/1) ...lyplatform/ALL_VALUES_ALL
2016-05-18 00:57:52,429 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:52,429 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533072290_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001_wd/localmode-log-dir/dag_1463533072290_0001_1.dot
2016-05-18 00:57:52,430 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001_wd/localmode-log-dir/dag_1463533072290_0001_1-tez-dag.pb.txt
2016-05-18 00:57:52,436 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533072290_0001_1
2016-05-18 00:57:52,436 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,436 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864] every/(1/1) ...lyplatform/ALL_VALUES_ALL
2016-05-18 00:57:52 Running Dag: dag_1463533072290_0001_1
2016-05-18 00:57:52,445 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533072290_0001_1, submitTime=1463533072429
2016-05-18 00:57:52,445 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001/tez-conf.pb" } size: 36913 timestamp: 1463533072000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:52,446 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,446 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:52,446 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001/tez-conf.pb" } size: 36913 timestamp: 1463533072000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:52,446 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,446 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:52,447 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:52,447 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533072290_0001_1, initTime=1463533072447
2016-05-18 00:57:52,447 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533072290_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:52,447 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:52,447 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:DAG_STARTED]: dagID=dag_1463533072290_0001_1, startTime=1463533072447
2016-05-18 00:57:52,447 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533072290_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:52,448 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...lyplatform/ALL_VALUES_ALL, applicationId=application_1463533072290_0001, dagName=[5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864] every/(1/1) ...lyplatform/ALL_VALUES_ALL
2016-05-18 00:57:52,448 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: C04827F4E02346358D122F759952B816 : {4F23A8D5DD36411C80D3C762FBD832A4={InputName=4F23A8D5DD36411C80D3C762FBD832A4}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:52,448 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 4F23A8D5DD36411C80D3C762FBD832A4, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:52,448 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,448 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,448 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,448 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,449 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:52,449 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,450 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 4F23A8D5DD36411C80D3C762FBD832A4 on vertex vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,451 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 4F23A8D5DD36411C80D3C762FBD832A4 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:52,452 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:52,452 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,452 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,453 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533072290_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:52,453 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235], numAdditionalOutputs=1
2016-05-18 00:57:52,453 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=ADDEEB049DC34A2F91978E4FF03B4376, vertex=vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:52,454 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:52,454 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:52,454 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:52,454 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:52,455 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:52,455 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:52,455 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for F0F67A2AC1BD4A05B966D3229ADC2235:ADDEEB049DC34A2F91978E4FF03B4376 using oldmapred API
2016-05-18 00:57:52,455 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=ADDEEB049DC34A2F91978E4FF03B4376, vertexName=F0F67A2AC1BD4A05B966D3229ADC2235, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:52,456 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 4F23A8D5DD36411C80D3C762FBD832A4 on vertex vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,456 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F0F67A2AC1BD4A05B966D3229ADC2235, vertexId=vertex_1463533072290_0001_1_01, initRequestedTime=1463533072449, initedTime=1463533072453, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:52,456 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:52,457 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:52,457 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {4F23A8D5DD36411C80D3C762FBD832A4=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:52,457 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] parallelism set to 1
2016-05-18 00:57:52,458 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:52,458 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C04827F4E02346358D122F759952B816, vertexId=vertex_1463533072290_0001_1_00, initRequestedTime=1463533072448, initedTime=1463533072458, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:52,458 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] with distanceFromRoot: 0
2016-05-18 00:57:52,458 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:52,458 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533072290_0001_1_00, startRequestedTime=1463533072449, startedTime=1463533072458
2016-05-18 00:57:52,458 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533072290_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:52,458 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:52,458 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533072290_0001_1_00 for vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] numStartedSources: 1 numSources: 1
2016-05-18 00:57:52,459 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] with distanceFromRoot: 1
2016-05-18 00:57:52,459 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533072290_0001_1_01, startRequestedTime=1463533072458, startedTime=1463533072459
2016-05-18 00:57:52,459 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:52,460 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in C04827F4E02346358D122F759952B816
2016-05-18 00:57:52,460 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,460 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: F0F67A2AC1BD4A05B966D3229ADC2235 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:52,460 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:TASK_STARTED]: vertexName=C04827F4E02346358D122F759952B816, taskId=task_1463533072290_0001_1_00_000000, scheduledTime=1463533072460, launchTime=1463533072460
2016-05-18 00:57:52,461 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,461 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: C04827F4E02346358D122F759952B816 in vertex: F0F67A2AC1BD4A05B966D3229ADC2235 numBipartiteSourceTasks: 1
2016-05-18 00:57:52,462 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:52,466 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533072290_0001_00_000001, launchTime=1463533072466
2016-05-18 00:57:52,467 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533072290_0001_00_000001
2016-05-18 00:57:52,467 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533072290_0001_1_00_000000_0] started. Is using containerId: [container_1463533072290_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:52,467 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C04827F4E02346358D122F759952B816, taskAttemptId=attempt_1463533072290_0001_1_00_000000_0, startTime=1463533072467, containerId=container_1463533072290_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:52,468 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533072290_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533072290_0001_1_00_000000_0
2016-05-18 00:57:52 Starting to run new task attempt: attempt_1463533072290_0001_1_00_000000_0
2016-05-18 00:57:52,469 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533072290_0001_1_00_000000_0
2016-05-18 00:57:52,469 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,469 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:52,469 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533072290_0001, with dag id: dag_1463533072290_0001_1
2016-05-18 00:57:52,469 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864] every/(1/1) ...lyplatform/ALL_VALUES_ALL, VertexName: C04827F4E02346358D122F759952B816, VertexParallelism: 1, TaskAttemptID:attempt_1463533072290_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=4F23A8D5DD36411C80D3C762FBD832A4, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F0F67A2AC1BD4A05B966D3229ADC2235, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:52,470 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:52,470 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533072290_0001_1_00_000000_0
2016-05-18 00:57:52,472 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533072290_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:52,472 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533072290_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:52,473 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:52,473 INFO input.MRInput (MRInput.java:initialize(440)) - 4F23A8D5DD36411C80D3C762FBD832A4 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:52,474 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 4F23A8D5DD36411C80D3C762FBD832A4
2016-05-18 00:57:52,477 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C04827F4E02346358D122F759952B816, ordinal: 0
2016-05-18 00:57:52,477 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C04827F4E02346358D122F759952B816, mem on start (mb), free: 75, total: 368, max: 672
2016-05-18 00:57:52,477 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:52,477 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:52,477 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:52,477 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:52,477 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[4F23A8D5DD36411C80D3C762FBD832A4:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [F0F67A2AC1BD4A05B966D3229ADC2235:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:52,478 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:52,478 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:52,478 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:52,478 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[4F23A8D5DD36411C80D3C762FBD832A4:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [F0F67A2AC1BD4A05B966D3229ADC2235:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:52,478 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:52,478 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 4F23A8D5DD36411C80D3C762FBD832A4 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:52,478 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:52,479 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:52,478 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 4F23A8D5DD36411C80D3C762FBD832A4
2016-05-18 00:57:52,479 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:52,479 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533072290_0001_1_00_000000_0
2016-05-18 00:57:52,480 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:52,480 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 4F23A8D5DD36411C80D3C762FBD832A4 initialized RecordReader from event
2016-05-18 00:57:52,482 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:52,482 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 4F23A8D5DD36411C80D3C762FBD832A4
2016-05-18 00:57:52,482 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 5D5E6D71A7D64B8F80A9027559BC867F
2016-05-18 00:57:52,482 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 5D5E6D71A7D64B8F80A9027559BC867F
2016-05-18 00:57:52,483 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,487 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,487 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - F0F67A2AC1BD4A05B966D3229ADC2235 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@63ffcb27, comparator=cascading.tuple.hadoop.util.TupleComparator@67966ccf, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:52,487 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for F0F67A2AC1BD4A05B966D3229ADC2235: , UsingHashComparator=false
2016-05-18 00:57:52,508 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:52,508 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:52,508 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for F0F67A2AC1BD4A05B966D3229ADC2235: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:52,508 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - F0F67A2AC1BD4A05B966D3229ADC2235: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:52,509 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:52,509 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C04827F4E02346358D122F759952B816, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:52,512 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533072290_0001_1_00_000000_0
2016-05-18 00:57:52,512 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C04827F4E02346358D122F759952B816, mem on close (mb), free: 169, total: 368, max: 672
2016-05-18 00:57:52,512 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - F0F67A2AC1BD4A05B966D3229ADC2235: Starting flush of map output
2016-05-18 00:57:52,512 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - F0F67A2AC1BD4A05B966D3229ADC2235: Span0.length = 20, perItem = 6
2016-05-18 00:57:52,512 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - F0F67A2AC1BD4A05B966D3229ADC2235: done sorting span=0, length=20, time=0
2016-05-18 00:57:52,513 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - F0F67A2AC1BD4A05B966D3229ADC2235: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:52,514 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - F0F67A2AC1BD4A05B966D3229ADC2235: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001_wd/localmode-local-dir/output/attempt_1463533072290_0001_1_00_000000_0_10303_0/file.out
2016-05-18 00:57:52,517 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:52 Completed running task attempt: attempt_1463533072290_0001_1_00_000000_0
2016-05-18 00:57:52,517 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533072290_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:52,517 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533072290_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:52,518 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533072290_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533072509, Process_Duration=3, Process_End_Time=1463533072512, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:52,518 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533072290_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:52,518 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:52,518 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C04827F4E02346358D122F759952B816, taskAttemptId=attempt_1463533072290_0001_1_00_000000_0, creationTime=1463533072461, allocationTime=1463533072466, startTime=1463533072467, finishTime=1463533072518, timeTaken=51, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:52,518 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:52,518 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,519 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533072290_0001_1_00_000000_0
2016-05-18 00:57:52,519 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533072290_0001_00_000001
2016-05-18 00:57:52,519 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:TASK_FINISHED]: vertexName=C04827F4E02346358D122F759952B816, taskId=task_1463533072290_0001_1_00_000000, startTime=1463533072467, finishTime=1463533072519, timeTaken=52, status=SUCCEEDED, successfulAttemptID=attempt_1463533072290_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072509, Process_Duration=3, Process_End_Time=1463533072512, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:52,519 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:52,519 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816]
2016-05-18 00:57:52,520 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:VERTEX_FINISHED]: vertexName=C04827F4E02346358D122F759952B816, vertexId=vertex_1463533072290_0001_1_00, initRequestedTime=1463533072448, initedTime=1463533072458, startRequestedTime=1463533072449, startedTime=1463533072458, finishTime=1463533072519, timeTaken=61, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072509, Process_Duration=3, Process_End_Time=1463533072512, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533072467, firstTasksToStart=[ task_1463533072290_0001_1_00_000000 ], lastTaskFinishTime=1463533072518, lastTasksToFinish=[ task_1463533072290_0001_1_00_000000 ], minTaskDuration=51, maxTaskDuration=51, avgTaskDuration=51.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533072290_0001_1_00_000000 ], longestDurationTasks=[ task_1463533072290_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:52,520 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:52,520 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533072290_0001_00_000001, stoppedTime=1463533072520, exitStatus=0
2016-05-18 00:57:52,520 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] attempt: attempt_1463533072290_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:52,520 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533072290_0001_1_00 [C04827F4E02346358D122F759952B816] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:52,520 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:52,520 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F0F67A2AC1BD4A05B966D3229ADC2235
2016-05-18 00:57:52,521 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533072290_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:52,521 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=C04827F4E02346358D122F759952B816, destinationVertexName=F0F67A2AC1BD4A05B966D3229ADC2235. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:52,521 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,521 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:TASK_STARTED]: vertexName=F0F67A2AC1BD4A05B966D3229ADC2235, taskId=task_1463533072290_0001_1_01_000000, scheduledTime=1463533072521, launchTime=1463533072521
2016-05-18 00:57:52,521 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,523 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533072290_0001_00_000002, launchTime=1463533072523
2016-05-18 00:57:52,523 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533072290_0001_00_000002
2016-05-18 00:57:52,525 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533072290_0001_00_000002: 1 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533072290_0001_1_01_000000_0
2016-05-18 00:57:52,525 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533072290_0001_1_01_000000_0] started. Is using containerId: [container_1463533072290_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:52,525 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F0F67A2AC1BD4A05B966D3229ADC2235, taskAttemptId=attempt_1463533072290_0001_1_01_000000_0, startTime=1463533072525, containerId=container_1463533072290_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:52 Starting to run new task attempt: attempt_1463533072290_0001_1_01_000000_0
2016-05-18 00:57:52,525 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533072290_0001_1_01_000000_0
2016-05-18 00:57:52,525 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,526 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:52,526 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864] every/(1/1) ...lyplatform/ALL_VALUES_ALL, VertexName: F0F67A2AC1BD4A05B966D3229ADC2235, VertexParallelism: 1, TaskAttemptID:attempt_1463533072290_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C04827F4E02346358D122F759952B816, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=ADDEEB049DC34A2F91978E4FF03B4376, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:52,526 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:52,526 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533072290_0001_1_01_000000_0
2016-05-18 00:57:52,527 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533072290_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,527 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533072290_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:52,529 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:52,529 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533072290_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:52,529 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533072290_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:52,529 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533072290_0001_00_000001, Exiting
2016-05-18 00:57:52,529 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533072290_0001_00_000001
2016-05-18 00:57:52,530 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533072290_0001_00_000001
2016-05-18 00:57:52,530 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:52,530 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533072290_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:52,530 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533072290_0001_00_000001 completed successfully
2016-05-18 00:57:52,534 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F0F67A2AC1BD4A05B966D3229ADC2235, ordinal: 1
2016-05-18 00:57:52,534 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F0F67A2AC1BD4A05B966D3229ADC2235, mem on start (mb), free: 164, total: 368, max: 672
2016-05-18 00:57:52,534 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:52,534 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:52,534 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:52,539 INFO output.MROutput (MROutput.java:initialize(421)) - ADDEEB049DC34A2F91978E4FF03B4376: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:52,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:52,539 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[C04827F4E02346358D122F759952B816:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [ADDEEB049DC34A2F91978E4FF03B4376:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:52,539 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:52,539 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:52,539 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:52,539 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[C04827F4E02346358D122F759952B816:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [ADDEEB049DC34A2F91978E4FF03B4376:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:52,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:52,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: C04827F4E02346358D122F759952B816 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:52,539 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:52,540 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:52,540 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - C04827F4E02346358D122F759952B816: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:52,540 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - C04827F4E02346358D122F759952B816: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:52,541 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: C04827F4E02346358D122F759952B816 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:52,541 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: C04827F4E02346358D122F759952B816
2016-05-18 00:57:52,541 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:52,542 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533072290_0001_1_01_000000_0
2016-05-18 00:57:52,543 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C04827F4E02346358D122F759952B816: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:52,544 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:52,544 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 5D5E6D71A7D64B8F80A9027559BC867F
2016-05-18 00:57:52,545 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_ALL"], id: ADDEEB049DC34A2F91978E4FF03B4376
2016-05-18 00:57:52,545 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_ALL"]
2016-05-18 00:57:52,545 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 5D5E6D71A7D64B8F80A9027559BC867F, for 1 inputs
2016-05-18 00:57:52,545 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533072290_0001_1_00_000000_0_10303} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533072545, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:52,545 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : C04827F4E02346358D122F759952B816
2016-05-18 00:57:52,546 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:52,546 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: C04827F4E02346358D122F759952B816, wasInterrupted=false
2016-05-18 00:57:52,546 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:52,547 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:52,548 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,548 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001_wd/localmode-local-dir/output/attempt_1463533072290_0001_1_00_000000_0_10303/file.out, len=181, isLocal=true
2016-05-18 00:57:52,548 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:52,549 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:52,549 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:52,549 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:52,549 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : C04827F4E02346358D122F759952B816
2016-05-18 00:57:52,549 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F0F67A2AC1BD4A05B966D3229ADC2235, all 1 inputs ready in: 00:00:00.004
2016-05-18 00:57:52,550 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - C04827F4E02346358D122F759952B816: Shuffle Runner thread complete
2016-05-18 00:57:52,550 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,551 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - C04827F4E02346358D122F759952B816: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:52,552 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:52,552 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:52,553 INFO output.MROutput (MROutput.java:close(526)) - ADDEEB049DC34A2F91978E4FF03B4376 closed
2016-05-18 00:57:52,553 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533072290_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:52,553 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533072290_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:52,554 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533072290_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533072550, Process_Duration=3, Process_End_Time=1463533072553, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:52,554 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:52,554 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F0F67A2AC1BD4A05B966D3229ADC2235, mem on close (mb), free: 161, total: 368, max: 672
2016-05-18 00:57:52,554 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=F0F67A2AC1BD4A05B966D3229ADC2235, index=1, interruptedStatus=false
2016-05-18 00:57:52,555 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: C04827F4E02346358D122F759952B816
2016-05-18 00:57:52,555 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - C04827F4E02346358D122F759952B816: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:52,555 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=F0F67A2AC1BD4A05B966D3229ADC2235, sourceVertex=C04827F4E02346358D122F759952B816, interruptedStatus=true
2016-05-18 00:57:52,555 INFO output.MROutput (MROutput.java:close(526)) - ADDEEB049DC34A2F91978E4FF03B4376 closed
2016-05-18 00:57:52,555 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=F0F67A2AC1BD4A05B966D3229ADC2235, sourceVertex=ADDEEB049DC34A2F91978E4FF03B4376, interruptedStatus=true
2016-05-18 00:57:52,556 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533072290_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:52,556 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:52,556 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533072290_0001_00_000002
2016-05-18 00:57:52,556 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533072290_0001_00_000002
2016-05-18 00:57:52,556 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533072290_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:52,556 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533072290_0001_00_000002 completed but with errors
2016-05-18 00:57:52,556 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F0F67A2AC1BD4A05B966D3229ADC2235, taskAttemptId=attempt_1463533072290_0001_1_01_000000_0, creationTime=1463533072521, allocationTime=1463533072522, startTime=1463533072525, finishTime=1463533072556, timeTaken=31, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533072290_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=6, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533072550, Process_Duration=3, Process_End_Time=1463533072553, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:52,557 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533072290_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:52,557 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533072290_0001_00_000002, stoppedTime=1463533072557, exitStatus=1
2016-05-18 00:57:52,557 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533072290_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:52,557 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:TASK_FINISHED]: vertexName=F0F67A2AC1BD4A05B966D3229ADC2235, taskId=task_1463533072290_0001_1_01_000000, startTime=1463533072525, finishTime=1463533072557, timeTaken=32, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533072290_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:52,558 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] because task failed: task_1463533072290_0001_1_01_000000
2016-05-18 00:57:52,558 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:52,558 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:52,558 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:52,558 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,558 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=ADDEEB049DC34A2F91978E4FF03B4376, vertexId=vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,559 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:VERTEX_FINISHED]: vertexName=F0F67A2AC1BD4A05B966D3229ADC2235, vertexId=vertex_1463533072290_0001_1_01, initRequestedTime=1463533072449, initedTime=1463533072453, startRequestedTime=1463533072458, startedTime=1463533072459, finishTime=1463533072559, timeTaken=100, status=FAILED, diagnostics=Task failed, taskId=task_1463533072290_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533072290_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533072525, firstTasksToStart=[ task_1463533072290_0001_1_01_000000 ], lastTaskFinishTime=1463533072556, lastTasksToFinish=[ task_1463533072290_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:52,560 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:52,560 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533072290_0001_1_01_000000_0 failed: true container: container_1463533072290_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:52,560 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:52,560 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:52,560 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:52,561 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533072290_0001_1, queueSize=0
2016-05-18 00:57:52,561 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072290_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533072290_0001_1, startTime=1463533072447, finishTime=1463533072561, timeTaken=114, status=FAILED, diagnostics=Vertex failed, vertexName=F0F67A2AC1BD4A05B966D3229ADC2235, vertexId=vertex_1463533072290_0001_1_01, diagnostics=[Task failed, taskId=task_1463533072290_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533072290_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=210, AM_GC_TIME_MILLIS=8, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072509, Process_Duration=3, Process_End_Time=1463533072512, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:52,561 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533072290_0001_1_01 [F0F67A2AC1BD4A05B966D3229ADC2235]
2016-05-18 00:57:52,562 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533072290_0001_1 finished with state: FAILED
2016-05-18 00:57:52,562 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533072290_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:52 Completed Dag: dag_1463533072290_0001_1
2016-05-18 00:57:52,562 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533072290_0001_1, dagState=FAILED
2016-05-18 00:57:52,562 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533072290_0001_1_post
2016-05-18 00:57:52,562 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,562 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:52,562 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:52,562 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864] every/(1/1) ...lyplatform/ALL_VALUES_ALL, with id=dag_1463533072290_0001_1
2016-05-18 00:57:52,563 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864] every/(1/1) ...lyplatform/ALL_VALUES_ALL, with id=dag_1463533072290_0001_1
2016-05-18 00:57:52,565 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...lyplatform/ALL_VALUES_ALL, applicationId=application_1463533072290_0001
2016-05-18 00:57:52,567 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:52,567 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:52,567 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:52,567 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:52,567 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:52,567 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:52,568 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:52,568 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:52,568 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:52,568 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:52,568 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:52,569 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:52,569 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:52,570 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:52,573 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/5DEB75E97DCF49EDB4F3CC89DE4FAE97/A55B25E46FB74519930CDC538FDE3864/.tez/application_1463533072290_0001
2016-05-18 00:57:52,573 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:52,573 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...lyplatform/ALL_VALUES_ALL
2016-05-18 00:57:52,574 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:52,574 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_ALL/_temporary
2016-05-18 00:57:52,575 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.302
2016-05-18 00:57:52,575 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_VALUES_ALL
2016-05-18 00:57:52,605 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:52,606 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:52,643 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.005
2016-05-18 00:57:52,649 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 00:57:52,649 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:52,649 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:52,650 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:52,667 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:52,671 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_REPLACE
2016-05-18 00:57:52,715 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:52,715 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:52,715 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_REPLACE"]
2016-05-18 00:57:52,715 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:52,715 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:52,716 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:52,717 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...atform/ALL_VALUES_REPLACE
2016-05-18 00:57:52,717 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:52,730 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:52,735 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:52,735 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:52,735 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:52,744 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001 doesn't exist and is created
2016-05-18 00:57:52,758 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:52,758 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:52,759 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001_wd
2016-05-18 00:57:52,770 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533072735_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:52,770 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:52,770 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:52,772 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:52,772 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:52,772 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:52,772 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:52,772 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:52,774 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:52,774 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:52,774 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:52,774 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:52,775 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:52,775 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:52,775 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:52,775 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:52,775 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:52,775 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:52,775 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:52,775 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:52,776 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:52,777 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:52,777 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:52,777 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001_wd/localmode-log-dir
2016-05-18 00:57:52,777 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001_wd/localmode-log-dir/history.txt.appattempt_1463533072735_0001_000000, maxErrors=10
2016-05-18 00:57:52,778 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:52,778 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533072735_0001_000000, appSubmitTime=1463533072768, launchTime=1463533072769
2016-05-18 00:57:52,780 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:52,783 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:60838
2016-05-18 00:57:52,783 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533072735
2016-05-18 00:57:52,785 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533072735_0001_000000, startTime=1463533072785
2016-05-18 00:57:52,785 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:52,858 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:52,858 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:52,858 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...atform/ALL_VALUES_REPLACE, applicationId=application_1463533072735_0001, dagName=[245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC] every/(1/1) ...atform/ALL_VALUES_REPLACE
2016-05-18 00:57:52,873 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC] every/(1/1) ...atform/ALL_VALUES_REPLACE
2016-05-18 00:57:52,873 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:52,874 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533072735_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001_wd/localmode-log-dir/dag_1463533072735_0001_1.dot
2016-05-18 00:57:52,874 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001_wd/localmode-log-dir/dag_1463533072735_0001_1-tez-dag.pb.txt
2016-05-18 00:57:52,880 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533072735_0001_1
2016-05-18 00:57:52,880 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,880 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC] every/(1/1) ...atform/ALL_VALUES_REPLACE
2016-05-18 00:57:52 Running Dag: dag_1463533072735_0001_1
2016-05-18 00:57:52,889 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533072735_0001_1, submitTime=1463533072873
2016-05-18 00:57:52,890 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001/tez-conf.pb" } size: 36913 timestamp: 1463533072000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:52,890 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,890 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:52,890 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001/tez-conf.pb" } size: 36913 timestamp: 1463533072000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:52,890 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:52,891 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:52,891 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:52,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533072735_0001_1, initTime=1463533072891
2016-05-18 00:57:52,891 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533072735_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:52,891 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:52,891 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:DAG_STARTED]: dagID=dag_1463533072735_0001_1, startTime=1463533072891
2016-05-18 00:57:52,891 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533072735_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:52,892 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A2EBFEA79A8346F098BBA2027AA96AFD : {D091CAE60BD04595BBBE9F01D6BE3A95={InputName=D091CAE60BD04595BBBE9F01D6BE3A95}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:52,892 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: D091CAE60BD04595BBBE9F01D6BE3A95, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:52,892 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,892 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,892 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,892 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,893 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...atform/ALL_VALUES_REPLACE, applicationId=application_1463533072735_0001, dagName=[245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC] every/(1/1) ...atform/ALL_VALUES_REPLACE
2016-05-18 00:57:52,893 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:52,893 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:52,894 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: D091CAE60BD04595BBBE9F01D6BE3A95 on vertex vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,896 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input D091CAE60BD04595BBBE9F01D6BE3A95 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:52,896 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:52,896 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:52,896 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:52,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533072735_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:52,897 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5], numAdditionalOutputs=1
2016-05-18 00:57:52,897 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=24034EF837904ACB8B4695B472BA3DBF, vertex=vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:52,898 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:52,898 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:52,899 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:52,899 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:52,899 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for CB3104161A1142739D0D90581CB47DC5:24034EF837904ACB8B4695B472BA3DBF using oldmapred API
2016-05-18 00:57:52,900 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=24034EF837904ACB8B4695B472BA3DBF, vertexName=CB3104161A1142739D0D90581CB47DC5, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:52,900 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: D091CAE60BD04595BBBE9F01D6BE3A95 on vertex vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,900 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:VERTEX_INITIALIZED]: vertexName=CB3104161A1142739D0D90581CB47DC5, vertexId=vertex_1463533072735_0001_1_01, initRequestedTime=1463533072893, initedTime=1463533072897, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:52,901 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:52,901 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:52,901 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {D091CAE60BD04595BBBE9F01D6BE3A95=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:52,902 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] parallelism set to 1
2016-05-18 00:57:52,902 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:52,902 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A2EBFEA79A8346F098BBA2027AA96AFD, vertexId=vertex_1463533072735_0001_1_00, initRequestedTime=1463533072892, initedTime=1463533072902, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:52,902 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] with distanceFromRoot: 0
2016-05-18 00:57:52,902 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:52,902 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533072735_0001_1_00, startRequestedTime=1463533072893, startedTime=1463533072902
2016-05-18 00:57:52,902 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533072735_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:52,903 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:52,903 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533072735_0001_1_00 for vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] numStartedSources: 1 numSources: 1
2016-05-18 00:57:52,903 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] with distanceFromRoot: 1
2016-05-18 00:57:52,903 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533072735_0001_1_01, startRequestedTime=1463533072903, startedTime=1463533072903
2016-05-18 00:57:52,903 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:52,903 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A2EBFEA79A8346F098BBA2027AA96AFD
2016-05-18 00:57:52,904 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,904 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:TASK_STARTED]: vertexName=A2EBFEA79A8346F098BBA2027AA96AFD, taskId=task_1463533072735_0001_1_00_000000, scheduledTime=1463533072904, launchTime=1463533072904
2016-05-18 00:57:52,904 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,906 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:52,906 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: CB3104161A1142739D0D90581CB47DC5 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:52,907 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A2EBFEA79A8346F098BBA2027AA96AFD in vertex: CB3104161A1142739D0D90581CB47DC5 numBipartiteSourceTasks: 1
2016-05-18 00:57:52,908 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533072735_0001_00_000001, launchTime=1463533072908
2016-05-18 00:57:52,908 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533072735_0001_00_000001
2016-05-18 00:57:52,909 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533072735_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533072735_0001_1_00_000000_0
2016-05-18 00:57:52,909 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533072735_0001_1_00_000000_0] started. Is using containerId: [container_1463533072735_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:52,909 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A2EBFEA79A8346F098BBA2027AA96AFD, taskAttemptId=attempt_1463533072735_0001_1_00_000000_0, startTime=1463533072909, containerId=container_1463533072735_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:52 Starting to run new task attempt: attempt_1463533072735_0001_1_00_000000_0
2016-05-18 00:57:52,909 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533072735_0001_1_00_000000_0
2016-05-18 00:57:52,910 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,910 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:52,910 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC] every/(1/1) ...atform/ALL_VALUES_REPLACE, VertexName: A2EBFEA79A8346F098BBA2027AA96AFD, VertexParallelism: 1, TaskAttemptID:attempt_1463533072735_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D091CAE60BD04595BBBE9F01D6BE3A95, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=CB3104161A1142739D0D90581CB47DC5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:52,910 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:52,910 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533072735_0001_1_00_000000_0
2016-05-18 00:57:52,912 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533072735_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:52,912 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533072735_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:52,914 INFO input.MRInput (MRInput.java:initialize(440)) - D091CAE60BD04595BBBE9F01D6BE3A95 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:52,915 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:52,914 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533072735_0001, with dag id: dag_1463533072735_0001_1
2016-05-18 00:57:52,918 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: D091CAE60BD04595BBBE9F01D6BE3A95
2016-05-18 00:57:52,920 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A2EBFEA79A8346F098BBA2027AA96AFD, ordinal: 0
2016-05-18 00:57:52,920 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A2EBFEA79A8346F098BBA2027AA96AFD, mem on start (mb), free: 76, total: 368, max: 672
2016-05-18 00:57:52,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:52,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:52,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:52,921 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:52,921 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[D091CAE60BD04595BBBE9F01D6BE3A95:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [CB3104161A1142739D0D90581CB47DC5:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:52,922 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:52,922 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:52,922 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:52,922 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[D091CAE60BD04595BBBE9F01D6BE3A95:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [CB3104161A1142739D0D90581CB47DC5:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:52,922 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:52,922 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D091CAE60BD04595BBBE9F01D6BE3A95 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:52,923 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:52,923 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:52,923 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D091CAE60BD04595BBBE9F01D6BE3A95
2016-05-18 00:57:52,923 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:52,924 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533072735_0001_1_00_000000_0
2016-05-18 00:57:52,925 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:52,925 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - D091CAE60BD04595BBBE9F01D6BE3A95 initialized RecordReader from event
2016-05-18 00:57:52,926 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:52,926 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: D091CAE60BD04595BBBE9F01D6BE3A95
2016-05-18 00:57:52,926 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 5EDC4E23E16E49E99BC3329F480AA71B
2016-05-18 00:57:52,927 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 5EDC4E23E16E49E99BC3329F480AA71B
2016-05-18 00:57:52,927 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,929 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,930 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - CB3104161A1142739D0D90581CB47DC5 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@31da2502, comparator=cascading.tuple.hadoop.util.TupleComparator@68b3bfdf, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:52,930 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for CB3104161A1142739D0D90581CB47DC5: , UsingHashComparator=false
2016-05-18 00:57:52,953 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:52,953 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:52,954 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for CB3104161A1142739D0D90581CB47DC5: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:52,954 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - CB3104161A1142739D0D90581CB47DC5: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:52,954 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:52,955 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A2EBFEA79A8346F098BBA2027AA96AFD, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:57:52,956 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533072735_0001_1_00_000000_0
2016-05-18 00:57:52,957 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A2EBFEA79A8346F098BBA2027AA96AFD, mem on close (mb), free: 169, total: 369, max: 672
2016-05-18 00:57:52,957 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - CB3104161A1142739D0D90581CB47DC5: Starting flush of map output
2016-05-18 00:57:52,957 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - CB3104161A1142739D0D90581CB47DC5: Span0.length = 20, perItem = 6
2016-05-18 00:57:52,957 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - CB3104161A1142739D0D90581CB47DC5: done sorting span=0, length=20, time=0
2016-05-18 00:57:52,957 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - CB3104161A1142739D0D90581CB47DC5: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:52,959 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - CB3104161A1142739D0D90581CB47DC5: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001_wd/localmode-local-dir/output/attempt_1463533072735_0001_1_00_000000_0_10309_0/file.out
2016-05-18 00:57:52,962 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:52 Completed running task attempt: attempt_1463533072735_0001_1_00_000000_0
2016-05-18 00:57:52,962 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533072735_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:52,962 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533072735_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:52,963 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533072735_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533072955, Process_Duration=1, Process_End_Time=1463533072956, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:52,963 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533072735_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:52,963 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:52,963 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A2EBFEA79A8346F098BBA2027AA96AFD, taskAttemptId=attempt_1463533072735_0001_1_00_000000_0, creationTime=1463533072904, allocationTime=1463533072907, startTime=1463533072909, finishTime=1463533072963, timeTaken=54, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:52,963 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:52,963 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,964 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533072735_0001_1_00_000000_0
2016-05-18 00:57:52,964 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533072735_0001_00_000001
2016-05-18 00:57:52,964 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:TASK_FINISHED]: vertexName=A2EBFEA79A8346F098BBA2027AA96AFD, taskId=task_1463533072735_0001_1_00_000000, startTime=1463533072909, finishTime=1463533072964, timeTaken=55, status=SUCCEEDED, successfulAttemptID=attempt_1463533072735_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072955, Process_Duration=1, Process_End_Time=1463533072956, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:52,964 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:52,965 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD]
2016-05-18 00:57:52,965 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:VERTEX_FINISHED]: vertexName=A2EBFEA79A8346F098BBA2027AA96AFD, vertexId=vertex_1463533072735_0001_1_00, initRequestedTime=1463533072892, initedTime=1463533072902, startRequestedTime=1463533072893, startedTime=1463533072902, finishTime=1463533072965, timeTaken=63, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072955, Process_Duration=1, Process_End_Time=1463533072956, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533072909, firstTasksToStart=[ task_1463533072735_0001_1_00_000000 ], lastTaskFinishTime=1463533072963, lastTasksToFinish=[ task_1463533072735_0001_1_00_000000 ], minTaskDuration=54, maxTaskDuration=54, avgTaskDuration=54.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533072735_0001_1_00_000000 ], longestDurationTasks=[ task_1463533072735_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:52,965 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:52,965 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533072735_0001_00_000001, stoppedTime=1463533072965, exitStatus=0
2016-05-18 00:57:52,966 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] attempt: attempt_1463533072735_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:52,966 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533072735_0001_1_00 [A2EBFEA79A8346F098BBA2027AA96AFD] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:52,966 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:52,966 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533072735_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:52,966 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533072735_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:52,966 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: CB3104161A1142739D0D90581CB47DC5
2016-05-18 00:57:52,966 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A2EBFEA79A8346F098BBA2027AA96AFD, destinationVertexName=CB3104161A1142739D0D90581CB47DC5. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:52,966 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:52,967 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:TASK_STARTED]: vertexName=CB3104161A1142739D0D90581CB47DC5, taskId=task_1463533072735_0001_1_01_000000, scheduledTime=1463533072966, launchTime=1463533072966
2016-05-18 00:57:52,967 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:52,966 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533072735_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:52,968 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533072735_0001_00_000001, Exiting
2016-05-18 00:57:52,968 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533072735_0001_00_000001
2016-05-18 00:57:52,968 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533072735_0001_00_000001
2016-05-18 00:57:52,969 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533072735_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:52,969 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533072735_0001_00_000001 completed successfully
2016-05-18 00:57:52,969 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533072735_0001_00_000002, launchTime=1463533072969
2016-05-18 00:57:52,970 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533072735_0001_00_000002
2016-05-18 00:57:52,970 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533072735_0001_1_01_000000_0] started. Is using containerId: [container_1463533072735_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:52,970 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=CB3104161A1142739D0D90581CB47DC5, taskAttemptId=attempt_1463533072735_0001_1_01_000000_0, startTime=1463533072970, containerId=container_1463533072735_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:52,971 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533072735_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533072735_0001_1_01_000000_0
2016-05-18 00:57:52 Starting to run new task attempt: attempt_1463533072735_0001_1_01_000000_0
2016-05-18 00:57:52,972 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533072735_0001_1_01_000000_0
2016-05-18 00:57:52,972 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:52,972 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:52,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC] every/(1/1) ...atform/ALL_VALUES_REPLACE, VertexName: CB3104161A1142739D0D90581CB47DC5, VertexParallelism: 1, TaskAttemptID:attempt_1463533072735_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A2EBFEA79A8346F098BBA2027AA96AFD, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=24034EF837904ACB8B4695B472BA3DBF, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:52,972 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:52,973 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533072735_0001_1_01_000000_0
2016-05-18 00:57:52,973 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533072735_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:52,973 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533072735_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:52,975 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:52,976 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:52,976 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:52,976 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:52,979 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: CB3104161A1142739D0D90581CB47DC5, ordinal: 1
2016-05-18 00:57:52,979 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CB3104161A1142739D0D90581CB47DC5, mem on start (mb), free: 164, total: 369, max: 672
2016-05-18 00:57:52,979 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:52,979 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:52,979 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:52,983 INFO output.MROutput (MROutput.java:initialize(421)) - 24034EF837904ACB8B4695B472BA3DBF: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:52,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:52,984 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[24034EF837904ACB8B4695B472BA3DBF:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A2EBFEA79A8346F098BBA2027AA96AFD:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:52,984 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:52,984 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:52,984 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:52,984 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[24034EF837904ACB8B4695B472BA3DBF:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A2EBFEA79A8346F098BBA2027AA96AFD:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:52,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:52,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A2EBFEA79A8346F098BBA2027AA96AFD being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:52,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:52,984 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:52,984 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A2EBFEA79A8346F098BBA2027AA96AFD: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:52,985 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A2EBFEA79A8346F098BBA2027AA96AFD: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:52,986 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A2EBFEA79A8346F098BBA2027AA96AFD with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:52,986 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A2EBFEA79A8346F098BBA2027AA96AFD
2016-05-18 00:57:52,987 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:52,987 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533072735_0001_1_01_000000_0
2016-05-18 00:57:52,987 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A2EBFEA79A8346F098BBA2027AA96AFD: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:52,988 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533072735_0001_1_00_000000_0_10309} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533072988, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:52,989 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A2EBFEA79A8346F098BBA2027AA96AFD
2016-05-18 00:57:52,989 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:52,989 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A2EBFEA79A8346F098BBA2027AA96AFD, wasInterrupted=false
2016-05-18 00:57:52,989 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:52,989 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:52,990 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 5EDC4E23E16E49E99BC3329F480AA71B
2016-05-18 00:57:52,990 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_REPLACE"], id: 24034EF837904ACB8B4695B472BA3DBF
2016-05-18 00:57:52,990 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_REPLACE"]
2016-05-18 00:57:52,990 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 5EDC4E23E16E49E99BC3329F480AA71B, for 1 inputs
2016-05-18 00:57:52,990 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:52,992 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,992 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001_wd/localmode-local-dir/output/attempt_1463533072735_0001_1_00_000000_0_10309/file.out, len=181, isLocal=true
2016-05-18 00:57:52,993 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:52,993 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:52,993 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:52,993 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:52,993 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A2EBFEA79A8346F098BBA2027AA96AFD
2016-05-18 00:57:52,993 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A2EBFEA79A8346F098BBA2027AA96AFD: Shuffle Runner thread complete
2016-05-18 00:57:52,993 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: CB3104161A1142739D0D90581CB47DC5, all 1 inputs ready in: 00:00:00.003
2016-05-18 00:57:52,994 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:52,994 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A2EBFEA79A8346F098BBA2027AA96AFD: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:52,996 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:53,000 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:53,004 INFO output.MROutput (MROutput.java:close(526)) - 24034EF837904ACB8B4695B472BA3DBF closed
2016-05-18 00:57:53,004 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533072735_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:53,005 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533072735_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:53,005 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533072735_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533072994, Process_Duration=11, Process_End_Time=1463533073005, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:53,005 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:53,006 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: CB3104161A1142739D0D90581CB47DC5, mem on close (mb), free: 160, total: 369, max: 672
2016-05-18 00:57:53,006 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=CB3104161A1142739D0D90581CB47DC5, index=1, interruptedStatus=false
2016-05-18 00:57:53,006 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A2EBFEA79A8346F098BBA2027AA96AFD
2016-05-18 00:57:53,006 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A2EBFEA79A8346F098BBA2027AA96AFD: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:53,006 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=CB3104161A1142739D0D90581CB47DC5, sourceVertex=A2EBFEA79A8346F098BBA2027AA96AFD, interruptedStatus=true
2016-05-18 00:57:53,007 INFO output.MROutput (MROutput.java:close(526)) - 24034EF837904ACB8B4695B472BA3DBF closed
2016-05-18 00:57:53,007 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=CB3104161A1142739D0D90581CB47DC5, sourceVertex=24034EF837904ACB8B4695B472BA3DBF, interruptedStatus=true
2016-05-18 00:57:53,007 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533072735_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:53,007 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:53,007 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533072735_0001_00_000002
2016-05-18 00:57:53,008 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533072735_0001_00_000002
2016-05-18 00:57:53,008 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=CB3104161A1142739D0D90581CB47DC5, taskAttemptId=attempt_1463533072735_0001_1_01_000000_0, creationTime=1463533072967, allocationTime=1463533072968, startTime=1463533072970, finishTime=1463533073007, timeTaken=37, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533072735_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533072994, Process_Duration=11, Process_End_Time=1463533073005, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:53,008 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533072735_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:53,008 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533072735_0001_00_000002 completed but with errors
2016-05-18 00:57:53,008 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533072735_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:53,008 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:TASK_FINISHED]: vertexName=CB3104161A1142739D0D90581CB47DC5, taskId=task_1463533072735_0001_1_01_000000, startTime=1463533072970, finishTime=1463533073008, timeTaken=38, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533072735_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:53,008 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533072735_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:53,008 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533072735_0001_00_000002, stoppedTime=1463533073008, exitStatus=1
2016-05-18 00:57:53,009 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] because task failed: task_1463533072735_0001_1_01_000000
2016-05-18 00:57:53,009 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:53,009 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:53,009 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:53,009 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:53,009 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=24034EF837904ACB8B4695B472BA3DBF, vertexId=vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:53,010 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:VERTEX_FINISHED]: vertexName=CB3104161A1142739D0D90581CB47DC5, vertexId=vertex_1463533072735_0001_1_01, initRequestedTime=1463533072893, initedTime=1463533072897, startRequestedTime=1463533072903, startedTime=1463533072903, finishTime=1463533073010, timeTaken=107, status=FAILED, diagnostics=Task failed, taskId=task_1463533072735_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533072735_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533072970, firstTasksToStart=[ task_1463533072735_0001_1_01_000000 ], lastTaskFinishTime=1463533073007, lastTasksToFinish=[ task_1463533072735_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:53,011 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:53,011 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533072735_0001_1_01_000000_0 failed: true container: container_1463533072735_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:53,011 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:53,011 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:53,011 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:53,012 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533072735_0001_1, queueSize=0
2016-05-18 00:57:53,012 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533072735_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533072735_0001_1, startTime=1463533072891, finishTime=1463533073011, timeTaken=120, status=FAILED, diagnostics=Vertex failed, vertexName=CB3104161A1142739D0D90581CB47DC5, vertexId=vertex_1463533072735_0001_1_01, diagnostics=[Task failed, taskId=task_1463533072735_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533072735_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=200, AM_GC_TIME_MILLIS=7, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533072955, Process_Duration=1, Process_End_Time=1463533072956, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:53,012 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533072735_0001_1_01 [CB3104161A1142739D0D90581CB47DC5]
2016-05-18 00:57:53,012 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533072735_0001_1 finished with state: FAILED
2016-05-18 00:57:53,012 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533072735_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:53 Completed Dag: dag_1463533072735_0001_1
2016-05-18 00:57:53,013 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533072735_0001_1, dagState=FAILED
2016-05-18 00:57:53,013 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533072735_0001_1_post
2016-05-18 00:57:53,013 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,013 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:53,013 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:53,013 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC] every/(1/1) ...atform/ALL_VALUES_REPLACE, with id=dag_1463533072735_0001_1
2016-05-18 00:57:53,013 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC] every/(1/1) ...atform/ALL_VALUES_REPLACE, with id=dag_1463533072735_0001_1
2016-05-18 00:57:53,018 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...atform/ALL_VALUES_REPLACE, applicationId=application_1463533072735_0001
2016-05-18 00:57:53,020 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:53,020 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:53,020 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:53,020 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:53,020 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:53,020 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:53,020 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:53,021 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:53,021 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:53,021 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:53,021 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:53,022 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:53,022 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:53,023 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:53,025 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/245830C79DE947CC97829218D274E181/DA71138F5223439F9E8C9FAC7D6A2AEC/.tez/application_1463533072735_0001
2016-05-18 00:57:53,025 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:53,025 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...atform/ALL_VALUES_REPLACE
2016-05-18 00:57:53,026 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:53,026 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_REPLACE/_temporary
2016-05-18 00:57:53,026 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.311
2016-05-18 00:57:53,027 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_VALUES_REPLACE
2016-05-18 00:57:53,049 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:53,049 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:53,078 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:53,082 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.010
2016-05-18 00:57:53,082 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:53,082 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:53,082 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:53,103 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:53,107 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_SWAP
2016-05-18 00:57:53,153 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:53,153 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:53,153 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_SWAP"]
2016-05-18 00:57:53,153 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:53,153 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:53,153 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:53,154 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...yplatform/ALL_VALUES_SWAP
2016-05-18 00:57:53,154 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:53,165 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:53,170 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:53,170 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:53,170 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:53,179 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001 doesn't exist and is created
2016-05-18 00:57:53,193 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:53,193 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:53,193 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001_wd
2016-05-18 00:57:53,202 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533073170_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:53,202 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:53,202 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:53,204 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:53,204 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:53,204 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:53,204 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:53,204 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:53,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:53,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:53,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:53,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:53,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:53,205 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:53,205 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:53,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:53,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:53,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:53,206 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:53,206 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:53,207 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:53,208 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:53,208 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:53,208 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001_wd/localmode-log-dir
2016-05-18 00:57:53,208 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001_wd/localmode-log-dir/history.txt.appattempt_1463533073170_0001_000000, maxErrors=10
2016-05-18 00:57:53,208 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:53,208 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533073170_0001_000000, appSubmitTime=1463533073202, launchTime=1463533073202
2016-05-18 00:57:53,211 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:53,214 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:53319
2016-05-18 00:57:53,214 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533073170
2016-05-18 00:57:53,216 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533073170_0001_000000, startTime=1463533073216
2016-05-18 00:57:53,216 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:53,293 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:53,293 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:53,293 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...yplatform/ALL_VALUES_SWAP, applicationId=application_1463533073170_0001, dagName=[33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2] every/(1/1) ...yplatform/ALL_VALUES_SWAP
2016-05-18 00:57:53,310 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2] every/(1/1) ...yplatform/ALL_VALUES_SWAP
2016-05-18 00:57:53,310 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:53,311 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533073170_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001_wd/localmode-log-dir/dag_1463533073170_0001_1.dot
2016-05-18 00:57:53,311 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001_wd/localmode-log-dir/dag_1463533073170_0001_1-tez-dag.pb.txt
2016-05-18 00:57:53,320 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533073170_0001_1
2016-05-18 00:57:53,320 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,320 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2] every/(1/1) ...yplatform/ALL_VALUES_SWAP
2016-05-18 00:57:53 Running Dag: dag_1463533073170_0001_1
2016-05-18 00:57:53,329 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533073170_0001_1, submitTime=1463533073310
2016-05-18 00:57:53,330 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001/tez-conf.pb" } size: 36913 timestamp: 1463533073000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:53,330 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,331 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:53,331 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001/tez-conf.pb" } size: 36913 timestamp: 1463533073000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:53,331 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,331 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:53,332 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:53,332 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533073170_0001_1, initTime=1463533073332
2016-05-18 00:57:53,332 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533073170_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:53,332 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:53,333 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:DAG_STARTED]: dagID=dag_1463533073170_0001_1, startTime=1463533073332
2016-05-18 00:57:53,333 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533073170_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:53,333 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...yplatform/ALL_VALUES_SWAP, applicationId=application_1463533073170_0001, dagName=[33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2] every/(1/1) ...yplatform/ALL_VALUES_SWAP
2016-05-18 00:57:53,333 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A74B6A14C8A5413688462EC4C3537D3A : {91B20829FF9F4BAD95BC8F9AE76C82B1={InputName=91B20829FF9F4BAD95BC8F9AE76C82B1}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:53,333 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 91B20829FF9F4BAD95BC8F9AE76C82B1, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:53,333 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,333 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,333 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,333 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,334 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:53,334 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,335 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 91B20829FF9F4BAD95BC8F9AE76C82B1 on vertex vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,337 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:53,337 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,337 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,338 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 91B20829FF9F4BAD95BC8F9AE76C82B1 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:53,338 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533073170_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:53,338 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA], numAdditionalOutputs=1
2016-05-18 00:57:53,339 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=E84AAFD92315405A91654BDAA8D58D4A, vertex=vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:53,340 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:53,340 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:53,340 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:53,341 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:53,341 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for DA8AAC2525AA45C3AF6291D3EB4BB3BA:E84AAFD92315405A91654BDAA8D58D4A using oldmapred API
2016-05-18 00:57:53,341 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=E84AAFD92315405A91654BDAA8D58D4A, vertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:53,341 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 91B20829FF9F4BAD95BC8F9AE76C82B1 on vertex vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,342 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, vertexId=vertex_1463533073170_0001_1_01, initRequestedTime=1463533073334, initedTime=1463533073338, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:53,342 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:53,343 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:53,343 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {91B20829FF9F4BAD95BC8F9AE76C82B1=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:53,344 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] parallelism set to 1
2016-05-18 00:57:53,344 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:53,345 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A74B6A14C8A5413688462EC4C3537D3A, vertexId=vertex_1463533073170_0001_1_00, initRequestedTime=1463533073333, initedTime=1463533073344, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:53,345 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] with distanceFromRoot: 0
2016-05-18 00:57:53,345 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:53,346 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533073170_0001_1_00, startRequestedTime=1463533073334, startedTime=1463533073345
2016-05-18 00:57:53,346 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533073170_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:53,346 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:53,347 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533073170_0001_1_00 for vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] numStartedSources: 1 numSources: 1
2016-05-18 00:57:53,347 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] with distanceFromRoot: 1
2016-05-18 00:57:53,347 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A74B6A14C8A5413688462EC4C3537D3A
2016-05-18 00:57:53,348 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,348 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533073170_0001_1_01, startRequestedTime=1463533073347, startedTime=1463533073347
2016-05-18 00:57:53,348 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:53,349 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: DA8AAC2525AA45C3AF6291D3EB4BB3BA with 0 source tasks and 1 pending tasks
2016-05-18 00:57:53,350 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A74B6A14C8A5413688462EC4C3537D3A in vertex: DA8AAC2525AA45C3AF6291D3EB4BB3BA numBipartiteSourceTasks: 1
2016-05-18 00:57:53,349 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:TASK_STARTED]: vertexName=A74B6A14C8A5413688462EC4C3537D3A, taskId=task_1463533073170_0001_1_00_000000, scheduledTime=1463533073349, launchTime=1463533073349
2016-05-18 00:57:53,350 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,350 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533073170_0001, with dag id: dag_1463533073170_0001_1
2016-05-18 00:57:53,352 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:53,354 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533073170_0001_00_000001, launchTime=1463533073354
2016-05-18 00:57:53,355 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533073170_0001_00_000001
2016-05-18 00:57:53,355 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533073170_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533073170_0001_1_00_000000_0
2016-05-18 00:57:53,355 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533073170_0001_1_00_000000_0] started. Is using containerId: [container_1463533073170_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:53,355 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A74B6A14C8A5413688462EC4C3537D3A, taskAttemptId=attempt_1463533073170_0001_1_00_000000_0, startTime=1463533073355, containerId=container_1463533073170_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:53 Starting to run new task attempt: attempt_1463533073170_0001_1_00_000000_0
2016-05-18 00:57:53,356 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533073170_0001_1_00_000000_0
2016-05-18 00:57:53,356 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,356 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:53,356 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2] every/(1/1) ...yplatform/ALL_VALUES_SWAP, VertexName: A74B6A14C8A5413688462EC4C3537D3A, VertexParallelism: 1, TaskAttemptID:attempt_1463533073170_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=91B20829FF9F4BAD95BC8F9AE76C82B1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:53,356 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:53,358 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533073170_0001_1_00_000000_0
2016-05-18 00:57:53,358 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533073170_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:53,359 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533073170_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:53,361 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:53,363 INFO input.MRInput (MRInput.java:initialize(440)) - 91B20829FF9F4BAD95BC8F9AE76C82B1 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:53,364 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 91B20829FF9F4BAD95BC8F9AE76C82B1
2016-05-18 00:57:53,365 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A74B6A14C8A5413688462EC4C3537D3A, ordinal: 0
2016-05-18 00:57:53,365 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A74B6A14C8A5413688462EC4C3537D3A, mem on start (mb), free: 75, total: 369, max: 672
2016-05-18 00:57:53,365 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:53,365 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:53,365 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:53,365 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:53,365 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[DA8AAC2525AA45C3AF6291D3EB4BB3BA:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [91B20829FF9F4BAD95BC8F9AE76C82B1:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 00:57:53,366 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:53,366 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:53,366 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:53,366 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[DA8AAC2525AA45C3AF6291D3EB4BB3BA:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [91B20829FF9F4BAD95BC8F9AE76C82B1:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 00:57:53,367 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:53,367 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 91B20829FF9F4BAD95BC8F9AE76C82B1 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:53,367 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:53,367 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:53,367 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 91B20829FF9F4BAD95BC8F9AE76C82B1
2016-05-18 00:57:53,368 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:53,368 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533073170_0001_1_00_000000_0
2016-05-18 00:57:53,369 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:53,369 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 91B20829FF9F4BAD95BC8F9AE76C82B1 initialized RecordReader from event
2016-05-18 00:57:53,371 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:53,371 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 91B20829FF9F4BAD95BC8F9AE76C82B1
2016-05-18 00:57:53,371 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 20E8BA287C0140A79A6D46235A61B87B
2016-05-18 00:57:53,371 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 20E8BA287C0140A79A6D46235A61B87B
2016-05-18 00:57:53,372 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:53,374 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:53,374 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - DA8AAC2525AA45C3AF6291D3EB4BB3BA using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@49a63740, comparator=cascading.tuple.hadoop.util.TupleComparator@3f757086, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:53,375 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for DA8AAC2525AA45C3AF6291D3EB4BB3BA: , UsingHashComparator=false
2016-05-18 00:57:53,395 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:53,396 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:53,396 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for DA8AAC2525AA45C3AF6291D3EB4BB3BA: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:53,396 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - DA8AAC2525AA45C3AF6291D3EB4BB3BA: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:53,396 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:53,397 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A74B6A14C8A5413688462EC4C3537D3A, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:53,399 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533073170_0001_1_00_000000_0
2016-05-18 00:57:53,399 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A74B6A14C8A5413688462EC4C3537D3A, mem on close (mb), free: 159, total: 361, max: 672
2016-05-18 00:57:53,399 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - DA8AAC2525AA45C3AF6291D3EB4BB3BA: Starting flush of map output
2016-05-18 00:57:53,399 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - DA8AAC2525AA45C3AF6291D3EB4BB3BA: Span0.length = 20, perItem = 6
2016-05-18 00:57:53,399 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - DA8AAC2525AA45C3AF6291D3EB4BB3BA: done sorting span=0, length=20, time=0
2016-05-18 00:57:53,400 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - DA8AAC2525AA45C3AF6291D3EB4BB3BA: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:53,400 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - DA8AAC2525AA45C3AF6291D3EB4BB3BA: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001_wd/localmode-local-dir/output/attempt_1463533073170_0001_1_00_000000_0_10314_0/file.out
2016-05-18 00:57:53,404 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:53 Completed running task attempt: attempt_1463533073170_0001_1_00_000000_0
2016-05-18 00:57:53,404 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533073170_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:53,404 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533073170_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:53,404 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533073170_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533073397, Process_Duration=2, Process_End_Time=1463533073399, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:53,405 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533073170_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:53,405 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:53,405 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A74B6A14C8A5413688462EC4C3537D3A, taskAttemptId=attempt_1463533073170_0001_1_00_000000_0, creationTime=1463533073350, allocationTime=1463533073353, startTime=1463533073355, finishTime=1463533073405, timeTaken=50, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:53,405 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:53,405 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,405 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533073170_0001_00_000001
2016-05-18 00:57:53,405 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533073170_0001_1_00_000000_0
2016-05-18 00:57:53,406 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:TASK_FINISHED]: vertexName=A74B6A14C8A5413688462EC4C3537D3A, taskId=task_1463533073170_0001_1_00_000000, startTime=1463533073355, finishTime=1463533073405, timeTaken=50, status=SUCCEEDED, successfulAttemptID=attempt_1463533073170_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533073397, Process_Duration=2, Process_End_Time=1463533073399, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:53,406 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:53,406 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A]
2016-05-18 00:57:53,406 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:VERTEX_FINISHED]: vertexName=A74B6A14C8A5413688462EC4C3537D3A, vertexId=vertex_1463533073170_0001_1_00, initRequestedTime=1463533073333, initedTime=1463533073344, startRequestedTime=1463533073334, startedTime=1463533073345, finishTime=1463533073406, timeTaken=61, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533073397, Process_Duration=2, Process_End_Time=1463533073399, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533073355, firstTasksToStart=[ task_1463533073170_0001_1_00_000000 ], lastTaskFinishTime=1463533073405, lastTasksToFinish=[ task_1463533073170_0001_1_00_000000 ], minTaskDuration=50, maxTaskDuration=50, avgTaskDuration=50.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533073170_0001_1_00_000000 ], longestDurationTasks=[ task_1463533073170_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:53,407 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:53,407 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533073170_0001_00_000001, stoppedTime=1463533073407, exitStatus=0
2016-05-18 00:57:53,407 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] attempt: attempt_1463533073170_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:53,407 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533073170_0001_1_00 [A74B6A14C8A5413688462EC4C3537D3A] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:53,407 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:53,407 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DA8AAC2525AA45C3AF6291D3EB4BB3BA
2016-05-18 00:57:53,407 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533073170_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:53,407 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A74B6A14C8A5413688462EC4C3537D3A, destinationVertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:53,408 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,408 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:TASK_STARTED]: vertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, taskId=task_1463533073170_0001_1_01_000000, scheduledTime=1463533073408, launchTime=1463533073408
2016-05-18 00:57:53,408 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,409 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533073170_0001_00_000002, launchTime=1463533073409
2016-05-18 00:57:53,410 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533073170_0001_00_000002
2016-05-18 00:57:53,411 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533073170_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533073170_0001_1_01_000000_0
2016-05-18 00:57:53,411 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533073170_0001_1_01_000000_0] started. Is using containerId: [container_1463533073170_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:53,411 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, taskAttemptId=attempt_1463533073170_0001_1_01_000000_0, startTime=1463533073411, containerId=container_1463533073170_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:53 Starting to run new task attempt: attempt_1463533073170_0001_1_01_000000_0
2016-05-18 00:57:53,412 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533073170_0001_1_01_000000_0
2016-05-18 00:57:53,412 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,412 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:53,412 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2] every/(1/1) ...yplatform/ALL_VALUES_SWAP, VertexName: DA8AAC2525AA45C3AF6291D3EB4BB3BA, VertexParallelism: 1, TaskAttemptID:attempt_1463533073170_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A74B6A14C8A5413688462EC4C3537D3A, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=E84AAFD92315405A91654BDAA8D58D4A, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:53,413 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:53,413 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533073170_0001_1_01_000000_0
2016-05-18 00:57:53,413 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533073170_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,416 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:53,416 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533073170_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:53,416 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533073170_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:53,417 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533073170_0001_00_000001, Exiting
2016-05-18 00:57:53,417 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:53,417 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533073170_0001_00_000001
2016-05-18 00:57:53,417 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533073170_0001_00_000001
2016-05-18 00:57:53,417 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533073170_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:53,417 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533073170_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:53,417 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533073170_0001_00_000001 completed successfully
2016-05-18 00:57:53,422 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DA8AAC2525AA45C3AF6291D3EB4BB3BA, ordinal: 1
2016-05-18 00:57:53,423 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DA8AAC2525AA45C3AF6291D3EB4BB3BA, mem on start (mb), free: 154, total: 361, max: 672
2016-05-18 00:57:53,423 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:53,423 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:53,424 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:53,425 INFO output.MROutput (MROutput.java:initialize(421)) - E84AAFD92315405A91654BDAA8D58D4A: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:53,425 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:53,425 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[E84AAFD92315405A91654BDAA8D58D4A:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A74B6A14C8A5413688462EC4C3537D3A:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:53,425 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:53,425 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:53,426 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:53,426 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[E84AAFD92315405A91654BDAA8D58D4A:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A74B6A14C8A5413688462EC4C3537D3A:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:53,426 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:53,426 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A74B6A14C8A5413688462EC4C3537D3A being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:53,426 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A74B6A14C8A5413688462EC4C3537D3A: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:53,427 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A74B6A14C8A5413688462EC4C3537D3A: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:53,427 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:53,427 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:53,427 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A74B6A14C8A5413688462EC4C3537D3A with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:53,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A74B6A14C8A5413688462EC4C3537D3A
2016-05-18 00:57:53,428 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:53,428 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533073170_0001_1_01_000000_0
2016-05-18 00:57:53,429 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A74B6A14C8A5413688462EC4C3537D3A: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:53,430 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533073170_0001_1_00_000000_0_10314} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533073430, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:53,431 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A74B6A14C8A5413688462EC4C3537D3A
2016-05-18 00:57:53,431 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:53,431 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:53,431 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A74B6A14C8A5413688462EC4C3537D3A, wasInterrupted=false
2016-05-18 00:57:53,432 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:53,432 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 20E8BA287C0140A79A6D46235A61B87B
2016-05-18 00:57:53,433 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_SWAP"], id: E84AAFD92315405A91654BDAA8D58D4A
2016-05-18 00:57:53,433 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_SWAP"]
2016-05-18 00:57:53,434 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 20E8BA287C0140A79A6D46235A61B87B, for 1 inputs
2016-05-18 00:57:53,436 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:53,437 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:53,438 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001_wd/localmode-local-dir/output/attempt_1463533073170_0001_1_00_000000_0_10314/file.out, len=181, isLocal=true
2016-05-18 00:57:53,438 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:53,438 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:53,439 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:53,439 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:53,439 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A74B6A14C8A5413688462EC4C3537D3A
2016-05-18 00:57:53,440 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DA8AAC2525AA45C3AF6291D3EB4BB3BA, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:53,440 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A74B6A14C8A5413688462EC4C3537D3A: Shuffle Runner thread complete
2016-05-18 00:57:53,441 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:53,441 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A74B6A14C8A5413688462EC4C3537D3A: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:53,442 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:53,442 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:53,443 INFO output.MROutput (MROutput.java:close(526)) - E84AAFD92315405A91654BDAA8D58D4A closed
2016-05-18 00:57:53,443 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533073170_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:53,443 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533073170_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:53,443 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533073170_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533073440, Process_Duration=3, Process_End_Time=1463533073443, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:53,444 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:53,444 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DA8AAC2525AA45C3AF6291D3EB4BB3BA, mem on close (mb), free: 151, total: 361, max: 672
2016-05-18 00:57:53,444 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=DA8AAC2525AA45C3AF6291D3EB4BB3BA, index=1, interruptedStatus=false
2016-05-18 00:57:53,444 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A74B6A14C8A5413688462EC4C3537D3A
2016-05-18 00:57:53,444 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A74B6A14C8A5413688462EC4C3537D3A: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:53,444 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=DA8AAC2525AA45C3AF6291D3EB4BB3BA, sourceVertex=A74B6A14C8A5413688462EC4C3537D3A, interruptedStatus=true
2016-05-18 00:57:53,444 INFO output.MROutput (MROutput.java:close(526)) - E84AAFD92315405A91654BDAA8D58D4A closed
2016-05-18 00:57:53,444 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=DA8AAC2525AA45C3AF6291D3EB4BB3BA, sourceVertex=E84AAFD92315405A91654BDAA8D58D4A, interruptedStatus=true
2016-05-18 00:57:53,444 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533073170_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:53,444 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], containerShutdownRequested=false}
2016-05-18 00:57:53,445 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533073170_0001_00_000002
2016-05-18 00:57:53,445 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533073170_0001_00_000002
2016-05-18 00:57:53,445 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, taskAttemptId=attempt_1463533073170_0001_1_01_000000_0, creationTime=1463533073408, allocationTime=1463533073409, startTime=1463533073411, finishTime=1463533073444, timeTaken=33, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533073170_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533073440, Process_Duration=3, Process_End_Time=1463533073443, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:53,445 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533073170_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:53,445 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:TASK_FINISHED]: vertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, taskId=task_1463533073170_0001_1_01_000000, startTime=1463533073411, finishTime=1463533073445, timeTaken=34, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533073170_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:53,445 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533073170_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]'}
2016-05-18 00:57:53,445 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533073170_0001_00_000002 completed but with errors
2016-05-18 00:57:53,445 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] because task failed: task_1463533073170_0001_1_01_000000
2016-05-18 00:57:53,445 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:53,446 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:53,446 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:53,446 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,446 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=E84AAFD92315405A91654BDAA8D58D4A, vertexId=vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,447 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:VERTEX_FINISHED]: vertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, vertexId=vertex_1463533073170_0001_1_01, initRequestedTime=1463533073334, initedTime=1463533073338, startRequestedTime=1463533073347, startedTime=1463533073347, finishTime=1463533073447, timeTaken=100, status=FAILED, diagnostics=Task failed, taskId=task_1463533073170_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533073170_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533073411, firstTasksToStart=[ task_1463533073170_0001_1_01_000000 ], lastTaskFinishTime=1463533073444, lastTasksToFinish=[ task_1463533073170_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:53,447 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:53,447 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533073170_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
2016-05-18 00:57:53,448 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533073170_0001_00_000002, stoppedTime=1463533073448, exitStatus=1
2016-05-18 00:57:53,448 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533073170_0001_1_01_000000_0 failed: true container: container_1463533073170_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:53,448 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:53,448 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:53,448 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:53,449 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533073170_0001_1, queueSize=0
2016-05-18 00:57:53,449 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073170_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533073170_0001_1, startTime=1463533073332, finishTime=1463533073449, timeTaken=117, status=FAILED, diagnostics=Vertex failed, vertexName=DA8AAC2525AA45C3AF6291D3EB4BB3BA, vertexId=vertex_1463533073170_0001_1_01, diagnostics=[Task failed, taskId=task_1463533073170_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533073170_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:VALUES]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=210, AM_GC_TIME_MILLIS=8, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533073397, Process_Duration=2, Process_End_Time=1463533073399, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:53,451 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533073170_0001_1_01 [DA8AAC2525AA45C3AF6291D3EB4BB3BA]
2016-05-18 00:57:53,451 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533073170_0001_1 finished with state: FAILED
2016-05-18 00:57:53,452 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533073170_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:53 Completed Dag: dag_1463533073170_0001_1
2016-05-18 00:57:53,452 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533073170_0001_1, dagState=FAILED
2016-05-18 00:57:53,452 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533073170_0001_1_post
2016-05-18 00:57:53,452 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,453 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:53,453 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:53,453 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2] every/(1/1) ...yplatform/ALL_VALUES_SWAP, with id=dag_1463533073170_0001_1
2016-05-18 00:57:53,453 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2] every/(1/1) ...yplatform/ALL_VALUES_SWAP, with id=dag_1463533073170_0001_1
2016-05-18 00:57:53,456 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...yplatform/ALL_VALUES_SWAP, applicationId=application_1463533073170_0001
2016-05-18 00:57:53,457 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:53,458 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:53,458 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:53,458 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:53,458 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:53,458 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:53,458 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:53,458 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:53,459 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:53,459 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:53,459 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:53,459 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:53,459 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:53,459 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:53,461 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/33BCA55DEA944C3DAB543A4670F4D469/FDBB253C25254ADCADDA936433D76EE2/.tez/application_1463533073170_0001
2016-05-18 00:57:53,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:53,462 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...yplatform/ALL_VALUES_SWAP
2016-05-18 00:57:53,463 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:53,463 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_VALUES_SWAP/_temporary
2016-05-18 00:57:53,463 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.310
2016-05-18 00:57:53,463 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_VALUES_SWAP
2016-05-18 00:57:53,493 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:53,493 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:53,524 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:53,529 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 00:57:53,529 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:53,529 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:53,529 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:53,544 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:53,545 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:53,545 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:53,548 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_-1
2016-05-18 00:57:53,620 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:53,621 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:53,621 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_-1"]
2016-05-18 00:57:53,621 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:53,622 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:53,622 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:53,623 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...emblyplatform/ALL_ARGS_-1
2016-05-18 00:57:53,625 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:53,644 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:53,650 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:53,650 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:53,650 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:53,661 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001 doesn't exist and is created
2016-05-18 00:57:53,680 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:53,680 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:53,680 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001_wd
2016-05-18 00:57:53,691 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533073650_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:53,691 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:53,692 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:53,693 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:53,693 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:53,693 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:53,693 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:53,694 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:53,695 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:53,695 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:53,695 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:53,695 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:53,695 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:53,695 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:53,695 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:53,695 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:53,696 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:53,696 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:53,696 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:53,696 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:53,698 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:53,698 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:53,698 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:53,699 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001_wd/localmode-log-dir
2016-05-18 00:57:53,699 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001_wd/localmode-log-dir/history.txt.appattempt_1463533073650_0001_000000, maxErrors=10
2016-05-18 00:57:53,699 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:53,699 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533073650_0001_000000, appSubmitTime=1463533073691, launchTime=1463533073691
2016-05-18 00:57:53,702 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:53,705 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:43093
2016-05-18 00:57:53,705 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533073650
2016-05-18 00:57:53,706 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533073650_0001_000000, startTime=1463533073706
2016-05-18 00:57:53,707 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:53,780 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:53,780 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:53,780 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...emblyplatform/ALL_ARGS_-1, applicationId=application_1463533073650_0001, dagName=[9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016] every/(1/1) ...emblyplatform/ALL_ARGS_-1
2016-05-18 00:57:53,795 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016] every/(1/1) ...emblyplatform/ALL_ARGS_-1
2016-05-18 00:57:53,796 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:53,797 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533073650_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001_wd/localmode-log-dir/dag_1463533073650_0001_1.dot
2016-05-18 00:57:53,798 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001_wd/localmode-log-dir/dag_1463533073650_0001_1-tez-dag.pb.txt
2016-05-18 00:57:53,804 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533073650_0001_1
2016-05-18 00:57:53,804 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,805 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016] every/(1/1) ...emblyplatform/ALL_ARGS_-1
2016-05-18 00:57:53 Running Dag: dag_1463533073650_0001_1
2016-05-18 00:57:53,814 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533073650_0001_1, submitTime=1463533073796
2016-05-18 00:57:53,815 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001/tez-conf.pb" } size: 36913 timestamp: 1463533073000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:53,815 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,815 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:53,816 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001/tez-conf.pb" } size: 36913 timestamp: 1463533073000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:53,816 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,816 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:53,816 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:53,817 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533073650_0001_1, initTime=1463533073817
2016-05-18 00:57:53,817 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533073650_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:53,817 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:53,817 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:DAG_STARTED]: dagID=dag_1463533073650_0001_1, startTime=1463533073817
2016-05-18 00:57:53,817 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533073650_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:53,817 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: D7F74C9E90114F18AA2176CC282C503B : {88FCE253322847B58B4451EBD0D9878F={InputName=88FCE253322847B58B4451EBD0D9878F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:53,817 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 88FCE253322847B58B4451EBD0D9878F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:53,818 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,818 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,818 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,818 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,818 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:53,818 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,819 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 88FCE253322847B58B4451EBD0D9878F on vertex vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,821 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 88FCE253322847B58B4451EBD0D9878F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:53,821 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:53,822 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,822 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,822 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533073650_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:53,822 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0], numAdditionalOutputs=1
2016-05-18 00:57:53,822 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=6F5BE13E57324B0BAE1F027597288027, vertex=vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:53,822 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:53,823 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:53,823 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:53,823 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:53,823 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 88FCE253322847B58B4451EBD0D9878F on vertex vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,824 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A6A408F036FE4714949AFA41DB0C69B0:6F5BE13E57324B0BAE1F027597288027 using oldmapred API
2016-05-18 00:57:53,824 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=6F5BE13E57324B0BAE1F027597288027, vertexName=A6A408F036FE4714949AFA41DB0C69B0, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:53,824 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...emblyplatform/ALL_ARGS_-1, applicationId=application_1463533073650_0001, dagName=[9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016] every/(1/1) ...emblyplatform/ALL_ARGS_-1
2016-05-18 00:57:53,827 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A6A408F036FE4714949AFA41DB0C69B0, vertexId=vertex_1463533073650_0001_1_01, initRequestedTime=1463533073818, initedTime=1463533073822, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:53,827 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:53,828 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:53,828 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {88FCE253322847B58B4451EBD0D9878F=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:53,828 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] parallelism set to 1
2016-05-18 00:57:53,828 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:53,829 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D7F74C9E90114F18AA2176CC282C503B, vertexId=vertex_1463533073650_0001_1_00, initRequestedTime=1463533073817, initedTime=1463533073829, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:53,829 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] with distanceFromRoot: 0
2016-05-18 00:57:53,829 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:53,829 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533073650_0001_1_00, startRequestedTime=1463533073818, startedTime=1463533073829
2016-05-18 00:57:53,829 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533073650_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:53,829 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:53,829 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533073650_0001_1_00 for vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] numStartedSources: 1 numSources: 1
2016-05-18 00:57:53,829 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] with distanceFromRoot: 1
2016-05-18 00:57:53,830 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533073650_0001_1_01, startRequestedTime=1463533073829, startedTime=1463533073829
2016-05-18 00:57:53,830 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:53,830 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A6A408F036FE4714949AFA41DB0C69B0 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:53,831 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: D7F74C9E90114F18AA2176CC282C503B in vertex: A6A408F036FE4714949AFA41DB0C69B0 numBipartiteSourceTasks: 1
2016-05-18 00:57:53,833 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in D7F74C9E90114F18AA2176CC282C503B
2016-05-18 00:57:53,833 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,833 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:TASK_STARTED]: vertexName=D7F74C9E90114F18AA2176CC282C503B, taskId=task_1463533073650_0001_1_00_000000, scheduledTime=1463533073833, launchTime=1463533073833
2016-05-18 00:57:53,834 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,834 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:53,836 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533073650_0001_00_000001, launchTime=1463533073836
2016-05-18 00:57:53,836 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533073650_0001_00_000001
2016-05-18 00:57:53,837 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533073650_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533073650_0001_1_00_000000_0
2016-05-18 00:57:53,837 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533073650_0001_1_00_000000_0] started. Is using containerId: [container_1463533073650_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:53,837 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D7F74C9E90114F18AA2176CC282C503B, taskAttemptId=attempt_1463533073650_0001_1_00_000000_0, startTime=1463533073837, containerId=container_1463533073650_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:53 Starting to run new task attempt: attempt_1463533073650_0001_1_00_000000_0
2016-05-18 00:57:53,838 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533073650_0001_1_00_000000_0
2016-05-18 00:57:53,838 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,838 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:53,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016] every/(1/1) ...emblyplatform/ALL_ARGS_-1, VertexName: D7F74C9E90114F18AA2176CC282C503B, VertexParallelism: 1, TaskAttemptID:attempt_1463533073650_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=88FCE253322847B58B4451EBD0D9878F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A6A408F036FE4714949AFA41DB0C69B0, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:53,839 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:53,840 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533073650_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:53,840 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533073650_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:53,841 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533073650_0001_1_00_000000_0
2016-05-18 00:57:53,841 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533073650_0001, with dag id: dag_1463533073650_0001_1
2016-05-18 00:57:53,844 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:53,846 INFO input.MRInput (MRInput.java:initialize(440)) - 88FCE253322847B58B4451EBD0D9878F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:53,846 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 88FCE253322847B58B4451EBD0D9878F
2016-05-18 00:57:53,848 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D7F74C9E90114F18AA2176CC282C503B, ordinal: 0
2016-05-18 00:57:53,848 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D7F74C9E90114F18AA2176CC282C503B, mem on start (mb), free: 66, total: 361, max: 672
2016-05-18 00:57:53,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:53,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:53,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:53,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:53,848 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[88FCE253322847B58B4451EBD0D9878F:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [A6A408F036FE4714949AFA41DB0C69B0:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:53,849 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:53,849 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:53,849 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:53,849 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[88FCE253322847B58B4451EBD0D9878F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [A6A408F036FE4714949AFA41DB0C69B0:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:53,849 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:53,849 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 88FCE253322847B58B4451EBD0D9878F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:53,849 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:53,849 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:53,849 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 88FCE253322847B58B4451EBD0D9878F
2016-05-18 00:57:53,850 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:53,850 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533073650_0001_1_00_000000_0
2016-05-18 00:57:53,851 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:53,851 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 88FCE253322847B58B4451EBD0D9878F initialized RecordReader from event
2016-05-18 00:57:53,852 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:53,853 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 88FCE253322847B58B4451EBD0D9878F
2016-05-18 00:57:53,853 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 3C44E55DEC824C4FA9A162194C2C4A86
2016-05-18 00:57:53,853 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 3C44E55DEC824C4FA9A162194C2C4A86
2016-05-18 00:57:53,854 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:53,855 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:53,856 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A6A408F036FE4714949AFA41DB0C69B0 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@741b28d3, comparator=cascading.tuple.hadoop.util.TupleComparator@23157032, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:53,856 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A6A408F036FE4714949AFA41DB0C69B0: , UsingHashComparator=false
2016-05-18 00:57:53,876 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:53,876 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:53,876 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A6A408F036FE4714949AFA41DB0C69B0: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:53,876 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A6A408F036FE4714949AFA41DB0C69B0: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:53,877 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:53,877 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D7F74C9E90114F18AA2176CC282C503B, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:53,879 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533073650_0001_1_00_000000_0
2016-05-18 00:57:53,879 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D7F74C9E90114F18AA2176CC282C503B, mem on close (mb), free: 165, total: 369, max: 672
2016-05-18 00:57:53,880 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A6A408F036FE4714949AFA41DB0C69B0: Starting flush of map output
2016-05-18 00:57:53,880 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A6A408F036FE4714949AFA41DB0C69B0: Span0.length = 20, perItem = 6
2016-05-18 00:57:53,880 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A6A408F036FE4714949AFA41DB0C69B0: done sorting span=0, length=20, time=0
2016-05-18 00:57:53,880 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A6A408F036FE4714949AFA41DB0C69B0: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:53,882 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A6A408F036FE4714949AFA41DB0C69B0: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001_wd/localmode-local-dir/output/attempt_1463533073650_0001_1_00_000000_0_10321_0/file.out
2016-05-18 00:57:53,885 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:53 Completed running task attempt: attempt_1463533073650_0001_1_00_000000_0
2016-05-18 00:57:53,885 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533073650_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:53,886 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533073650_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:53,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533073650_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533073877, Process_Duration=2, Process_End_Time=1463533073879, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:53,886 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533073650_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:53,887 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:53,887 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:53,887 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,887 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D7F74C9E90114F18AA2176CC282C503B, taskAttemptId=attempt_1463533073650_0001_1_00_000000_0, creationTime=1463533073834, allocationTime=1463533073835, startTime=1463533073837, finishTime=1463533073887, timeTaken=50, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:53,887 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533073650_0001_1_00_000000_0
2016-05-18 00:57:53,888 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:TASK_FINISHED]: vertexName=D7F74C9E90114F18AA2176CC282C503B, taskId=task_1463533073650_0001_1_00_000000, startTime=1463533073837, finishTime=1463533073887, timeTaken=50, status=SUCCEEDED, successfulAttemptID=attempt_1463533073650_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533073877, Process_Duration=2, Process_End_Time=1463533073879, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:53,888 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533073650_0001_00_000001
2016-05-18 00:57:53,888 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:53,888 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B]
2016-05-18 00:57:53,889 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:VERTEX_FINISHED]: vertexName=D7F74C9E90114F18AA2176CC282C503B, vertexId=vertex_1463533073650_0001_1_00, initRequestedTime=1463533073817, initedTime=1463533073829, startRequestedTime=1463533073818, startedTime=1463533073829, finishTime=1463533073888, timeTaken=59, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533073877, Process_Duration=2, Process_End_Time=1463533073879, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533073837, firstTasksToStart=[ task_1463533073650_0001_1_00_000000 ], lastTaskFinishTime=1463533073887, lastTasksToFinish=[ task_1463533073650_0001_1_00_000000 ], minTaskDuration=50, maxTaskDuration=50, avgTaskDuration=50.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533073650_0001_1_00_000000 ], longestDurationTasks=[ task_1463533073650_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:53,889 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:53,889 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533073650_0001_00_000001, stoppedTime=1463533073889, exitStatus=0
2016-05-18 00:57:53,889 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] attempt: attempt_1463533073650_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:53,889 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533073650_0001_1_00 [D7F74C9E90114F18AA2176CC282C503B] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:53,889 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:53,889 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A6A408F036FE4714949AFA41DB0C69B0
2016-05-18 00:57:53,889 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533073650_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:53,889 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=D7F74C9E90114F18AA2176CC282C503B, destinationVertexName=A6A408F036FE4714949AFA41DB0C69B0. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:53,890 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,890 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:TASK_STARTED]: vertexName=A6A408F036FE4714949AFA41DB0C69B0, taskId=task_1463533073650_0001_1_01_000000, scheduledTime=1463533073890, launchTime=1463533073890
2016-05-18 00:57:53,890 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533073650_0001_00_000002, launchTime=1463533073891
2016-05-18 00:57:53,892 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533073650_0001_00_000002
2016-05-18 00:57:53,892 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533073650_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533073650_0001_1_01_000000_0
2016-05-18 00:57:53,893 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533073650_0001_1_01_000000_0] started. Is using containerId: [container_1463533073650_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:53,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A6A408F036FE4714949AFA41DB0C69B0, taskAttemptId=attempt_1463533073650_0001_1_01_000000_0, startTime=1463533073892, containerId=container_1463533073650_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:53 Starting to run new task attempt: attempt_1463533073650_0001_1_01_000000_0
2016-05-18 00:57:53,893 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533073650_0001_1_01_000000_0
2016-05-18 00:57:53,893 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,893 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:53,893 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016] every/(1/1) ...emblyplatform/ALL_ARGS_-1, VertexName: A6A408F036FE4714949AFA41DB0C69B0, VertexParallelism: 1, TaskAttemptID:attempt_1463533073650_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D7F74C9E90114F18AA2176CC282C503B, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=6F5BE13E57324B0BAE1F027597288027, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:53,894 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:53,894 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533073650_0001_1_01_000000_0
2016-05-18 00:57:53,895 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533073650_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,895 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533073650_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:53,896 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:53,897 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:53,898 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533073650_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:53,898 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533073650_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:53,898 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533073650_0001_00_000001, Exiting
2016-05-18 00:57:53,898 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533073650_0001_00_000001
2016-05-18 00:57:53,899 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533073650_0001_00_000001
2016-05-18 00:57:53,899 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533073650_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:53,899 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533073650_0001_00_000001 completed successfully
2016-05-18 00:57:53,900 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A6A408F036FE4714949AFA41DB0C69B0, ordinal: 1
2016-05-18 00:57:53,900 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A6A408F036FE4714949AFA41DB0C69B0, mem on start (mb), free: 160, total: 369, max: 672
2016-05-18 00:57:53,901 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:53,901 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:53,901 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:53,906 INFO output.MROutput (MROutput.java:initialize(421)) - 6F5BE13E57324B0BAE1F027597288027: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:53,906 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:53,906 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[6F5BE13E57324B0BAE1F027597288027:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [D7F74C9E90114F18AA2176CC282C503B:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:53,906 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:53,906 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:53,906 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:53,906 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[6F5BE13E57324B0BAE1F027597288027:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [D7F74C9E90114F18AA2176CC282C503B:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:53,906 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:53,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: D7F74C9E90114F18AA2176CC282C503B being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:53,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:53,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:53,907 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - D7F74C9E90114F18AA2176CC282C503B: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:53,907 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - D7F74C9E90114F18AA2176CC282C503B: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:53,908 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: D7F74C9E90114F18AA2176CC282C503B with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:53,908 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: D7F74C9E90114F18AA2176CC282C503B
2016-05-18 00:57:53,908 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:53,908 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533073650_0001_1_01_000000_0
2016-05-18 00:57:53,909 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D7F74C9E90114F18AA2176CC282C503B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:53,910 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533073650_0001_1_00_000000_0_10321} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533073910, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:53,910 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : D7F74C9E90114F18AA2176CC282C503B
2016-05-18 00:57:53,911 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:53,911 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:53,911 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 3C44E55DEC824C4FA9A162194C2C4A86
2016-05-18 00:57:53,911 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: D7F74C9E90114F18AA2176CC282C503B, wasInterrupted=false
2016-05-18 00:57:53,911 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_-1"], id: 6F5BE13E57324B0BAE1F027597288027
2016-05-18 00:57:53,912 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:53,912 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_-1"]
2016-05-18 00:57:53,912 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:53,912 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 3C44E55DEC824C4FA9A162194C2C4A86, for 1 inputs
2016-05-18 00:57:53,913 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - D7F74C9E90114F18AA2176CC282C503B: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:53,913 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:53,913 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001_wd/localmode-local-dir/output/attempt_1463533073650_0001_1_00_000000_0_10321/file.out, len=181, isLocal=true
2016-05-18 00:57:53,913 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:53,913 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:53,913 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:53,913 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:53,914 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : D7F74C9E90114F18AA2176CC282C503B
2016-05-18 00:57:53,914 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - D7F74C9E90114F18AA2176CC282C503B: Shuffle Runner thread complete
2016-05-18 00:57:53,914 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A6A408F036FE4714949AFA41DB0C69B0, all 1 inputs ready in: 00:00:00.002
2016-05-18 00:57:53,915 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:53,915 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - D7F74C9E90114F18AA2176CC282C503B: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:53,917 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:53,917 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:53,918 INFO output.MROutput (MROutput.java:close(526)) - 6F5BE13E57324B0BAE1F027597288027 closed
2016-05-18 00:57:53,918 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533073650_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:53,918 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533073650_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:53,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533073650_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533073914, Process_Duration=4, Process_End_Time=1463533073918, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:53,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:53,919 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A6A408F036FE4714949AFA41DB0C69B0, mem on close (mb), free: 157, total: 369, max: 672
2016-05-18 00:57:53,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=A6A408F036FE4714949AFA41DB0C69B0, index=1, interruptedStatus=false
2016-05-18 00:57:53,919 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: D7F74C9E90114F18AA2176CC282C503B
2016-05-18 00:57:53,919 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - D7F74C9E90114F18AA2176CC282C503B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:53,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=A6A408F036FE4714949AFA41DB0C69B0, sourceVertex=D7F74C9E90114F18AA2176CC282C503B, interruptedStatus=true
2016-05-18 00:57:53,919 INFO output.MROutput (MROutput.java:close(526)) - 6F5BE13E57324B0BAE1F027597288027 closed
2016-05-18 00:57:53,919 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=A6A408F036FE4714949AFA41DB0C69B0, sourceVertex=6F5BE13E57324B0BAE1F027597288027, interruptedStatus=true
2016-05-18 00:57:53,920 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533073650_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], containerShutdownRequested=false}
2016-05-18 00:57:53,920 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], containerShutdownRequested=false}
2016-05-18 00:57:53,920 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533073650_0001_00_000002
2016-05-18 00:57:53,920 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A6A408F036FE4714949AFA41DB0C69B0, taskAttemptId=attempt_1463533073650_0001_1_01_000000_0, creationTime=1463533073890, allocationTime=1463533073891, startTime=1463533073892, finishTime=1463533073920, timeTaken=28, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533073650_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=7, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533073914, Process_Duration=4, Process_End_Time=1463533073918, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:53,920 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533073650_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:53,921 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:TASK_FINISHED]: vertexName=A6A408F036FE4714949AFA41DB0C69B0, taskId=task_1463533073650_0001_1_01_000000, startTime=1463533073892, finishTime=1463533073920, timeTaken=28, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533073650_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:53,921 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] because task failed: task_1463533073650_0001_1_01_000000
2016-05-18 00:57:53,921 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:53,921 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:53,921 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:53,921 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,921 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=6F5BE13E57324B0BAE1F027597288027, vertexId=vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,920 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533073650_0001_00_000002
2016-05-18 00:57:53,922 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533073650_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]'}
2016-05-18 00:57:53,922 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533073650_0001_00_000002 completed but with errors
2016-05-18 00:57:53,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:VERTEX_FINISHED]: vertexName=A6A408F036FE4714949AFA41DB0C69B0, vertexId=vertex_1463533073650_0001_1_01, initRequestedTime=1463533073818, initedTime=1463533073822, startRequestedTime=1463533073829, startedTime=1463533073829, finishTime=1463533073922, timeTaken=93, status=FAILED, diagnostics=Task failed, taskId=task_1463533073650_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533073650_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533073892, firstTasksToStart=[ task_1463533073650_0001_1_01_000000 ], lastTaskFinishTime=1463533073920, lastTasksToFinish=[ task_1463533073650_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:53,923 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:53,923 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533073650_0001_00_000002, stoppedTime=1463533073923, exitStatus=-1000
2016-05-18 00:57:53,923 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533073650_0001_1_01_000000_0 failed: true container: container_1463533073650_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:53,923 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533073650_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
2016-05-18 00:57:53,923 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:53,923 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:53,923 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:53,925 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533073650_0001_1, queueSize=0
2016-05-18 00:57:53,925 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533073650_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533073650_0001_1, startTime=1463533073817, finishTime=1463533073924, timeTaken=107, status=FAILED, diagnostics=Vertex failed, vertexName=A6A408F036FE4714949AFA41DB0C69B0, vertexId=vertex_1463533073650_0001_1_01, diagnostics=[Task failed, taskId=task_1463533073650_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533073650_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=200, AM_GC_TIME_MILLIS=7, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533073877, Process_Duration=2, Process_End_Time=1463533073879, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:53,925 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533073650_0001_1_01 [A6A408F036FE4714949AFA41DB0C69B0]
2016-05-18 00:57:53,925 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533073650_0001_1 finished with state: FAILED
2016-05-18 00:57:53,926 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533073650_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:53,926 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533073650_0001_00_000002
2016-05-18 00:57:53 Completed Dag: dag_1463533073650_0001_1
2016-05-18 00:57:53,926 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533073650_0001_1, dagState=FAILED
2016-05-18 00:57:53,926 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533073650_0001_1_post
2016-05-18 00:57:53,926 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:53,926 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:53,926 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:53,926 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016] every/(1/1) ...emblyplatform/ALL_ARGS_-1, with id=dag_1463533073650_0001_1
2016-05-18 00:57:53,927 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016] every/(1/1) ...emblyplatform/ALL_ARGS_-1, with id=dag_1463533073650_0001_1
2016-05-18 00:57:53,930 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...emblyplatform/ALL_ARGS_-1, applicationId=application_1463533073650_0001
2016-05-18 00:57:53,932 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:53,932 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:53,932 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:53,932 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:53,932 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:53,933 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:53,933 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:53,933 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:53,934 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:53,934 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:53,934 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:53,934 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:53,934 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:53,934 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:53,937 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/9968670E35EA48A2B9811CC29D15AE7C/BC6FB95BDB874E9B9320ACA324165016/.tez/application_1463533073650_0001
2016-05-18 00:57:53,937 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:53,938 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...emblyplatform/ALL_ARGS_-1
2016-05-18 00:57:53,938 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:53,938 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_-1/_temporary
2016-05-18 00:57:53,938 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.318
2016-05-18 00:57:53,939 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_ARGS_-1
2016-05-18 00:57:53,960 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:53,960 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:53,985 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.002
2016-05-18 00:57:53,989 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.006
2016-05-18 00:57:53,990 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:53,991 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:53,991 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_ARGS_'field'
2016-05-18 00:57:54,012 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:54,013 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:54,041 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:54,046 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 00:57:54,047 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:54,047 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:54,047 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:54,061 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:54,063 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:54,063 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:54,066 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_RESULTS
2016-05-18 00:57:54,112 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:54,113 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:54,113 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_RESULTS"]
2016-05-18 00:57:54,113 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:54,114 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:54,115 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:54,115 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...platform/ALL_ARGS_RESULTS
2016-05-18 00:57:54,116 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:54,128 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:54,132 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:54,132 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:54,133 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:54,142 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001 doesn't exist and is created
2016-05-18 00:57:54,155 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:54,155 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:54,155 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001_wd
2016-05-18 00:57:54,165 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533074132_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:54,165 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:54,165 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:54,166 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:54,166 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:54,166 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:54,166 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:54,167 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:54,168 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:54,168 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:54,168 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:54,168 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:54,168 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:54,168 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:54,168 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:54,169 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:54,169 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:54,169 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:54,169 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:54,169 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:54,171 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:54,172 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:54,172 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:54,172 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001_wd/localmode-log-dir
2016-05-18 00:57:54,172 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001_wd/localmode-log-dir/history.txt.appattempt_1463533074132_0001_000000, maxErrors=10
2016-05-18 00:57:54,173 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:54,173 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533074132_0001_000000, appSubmitTime=1463533074164, launchTime=1463533074164
2016-05-18 00:57:54,175 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:54,178 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:55367
2016-05-18 00:57:54,178 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533074132
2016-05-18 00:57:54,180 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533074132_0001_000000, startTime=1463533074180
2016-05-18 00:57:54,180 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:54,256 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:54,256 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:54,256 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...platform/ALL_ARGS_RESULTS, applicationId=application_1463533074132_0001, dagName=[D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932] every/(1/1) ...platform/ALL_ARGS_RESULTS
2016-05-18 00:57:54,272 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932] every/(1/1) ...platform/ALL_ARGS_RESULTS
2016-05-18 00:57:54,272 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:54,272 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533074132_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001_wd/localmode-log-dir/dag_1463533074132_0001_1.dot
2016-05-18 00:57:54,273 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001_wd/localmode-log-dir/dag_1463533074132_0001_1-tez-dag.pb.txt
2016-05-18 00:57:54,279 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533074132_0001_1
2016-05-18 00:57:54,279 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,279 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932] every/(1/1) ...platform/ALL_ARGS_RESULTS
2016-05-18 00:57:54 Running Dag: dag_1463533074132_0001_1
2016-05-18 00:57:54,287 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533074132_0001_1, submitTime=1463533074272
2016-05-18 00:57:54,288 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001/tez-conf.pb" } size: 36913 timestamp: 1463533074000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:54,288 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,288 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:54,289 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001/tez-conf.pb" } size: 36913 timestamp: 1463533074000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:54,289 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,289 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:54,289 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:54,289 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533074132_0001_1, initTime=1463533074289
2016-05-18 00:57:54,289 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533074132_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:54,289 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:54,290 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:DAG_STARTED]: dagID=dag_1463533074132_0001_1, startTime=1463533074290
2016-05-18 00:57:54,290 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533074132_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:54,290 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: BB1DA71DD7CD40FFBDA267E54089867B : {308B7803825945CA8E18257C77401370={InputName=308B7803825945CA8E18257C77401370}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:54,290 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 308B7803825945CA8E18257C77401370, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:54,290 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,290 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,290 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,290 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,291 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:54,291 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,291 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 308B7803825945CA8E18257C77401370 on vertex vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,292 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...platform/ALL_ARGS_RESULTS, applicationId=application_1463533074132_0001, dagName=[D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932] every/(1/1) ...platform/ALL_ARGS_RESULTS
2016-05-18 00:57:54,294 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:54,294 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,294 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,294 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533074132_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:54,295 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE], numAdditionalOutputs=1
2016-05-18 00:57:54,295 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=EEB8E00981C044D19B7579FFD891C235, vertex=vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:54,296 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 308B7803825945CA8E18257C77401370 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:54,297 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for AF985E64153C490DB74D308541DE72AE:EEB8E00981C044D19B7579FFD891C235 using oldmapred API
2016-05-18 00:57:54,297 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=EEB8E00981C044D19B7579FFD891C235, vertexName=AF985E64153C490DB74D308541DE72AE, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:54,297 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:54,297 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:54,298 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:54,298 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:54,298 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 308B7803825945CA8E18257C77401370 on vertex vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,298 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AF985E64153C490DB74D308541DE72AE, vertexId=vertex_1463533074132_0001_1_01, initRequestedTime=1463533074291, initedTime=1463533074294, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:54,299 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:54,300 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:54,300 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {308B7803825945CA8E18257C77401370=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:54,300 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] parallelism set to 1
2016-05-18 00:57:54,300 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:54,301 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BB1DA71DD7CD40FFBDA267E54089867B, vertexId=vertex_1463533074132_0001_1_00, initRequestedTime=1463533074290, initedTime=1463533074301, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:54,301 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] with distanceFromRoot: 0
2016-05-18 00:57:54,301 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:54,301 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533074132_0001_1_00, startRequestedTime=1463533074291, startedTime=1463533074301
2016-05-18 00:57:54,301 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533074132_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:54,301 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:54,301 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533074132_0001_1_00 for vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] numStartedSources: 1 numSources: 1
2016-05-18 00:57:54,301 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] with distanceFromRoot: 1
2016-05-18 00:57:54,303 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533074132_0001_1_01, startRequestedTime=1463533074301, startedTime=1463533074301
2016-05-18 00:57:54,303 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:54,303 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: AF985E64153C490DB74D308541DE72AE with 0 source tasks and 1 pending tasks
2016-05-18 00:57:54,304 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: BB1DA71DD7CD40FFBDA267E54089867B in vertex: AF985E64153C490DB74D308541DE72AE numBipartiteSourceTasks: 1
2016-05-18 00:57:54,305 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in BB1DA71DD7CD40FFBDA267E54089867B
2016-05-18 00:57:54,305 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,306 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:TASK_STARTED]: vertexName=BB1DA71DD7CD40FFBDA267E54089867B, taskId=task_1463533074132_0001_1_00_000000, scheduledTime=1463533074305, launchTime=1463533074305
2016-05-18 00:57:54,306 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,308 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:54,309 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533074132_0001_00_000001, launchTime=1463533074309
2016-05-18 00:57:54,310 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533074132_0001_00_000001
2016-05-18 00:57:54,310 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533074132_0001_1_00_000000_0] started. Is using containerId: [container_1463533074132_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:54,311 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BB1DA71DD7CD40FFBDA267E54089867B, taskAttemptId=attempt_1463533074132_0001_1_00_000000_0, startTime=1463533074310, containerId=container_1463533074132_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:54,311 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533074132_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533074132_0001_1_00_000000_0
2016-05-18 00:57:54 Starting to run new task attempt: attempt_1463533074132_0001_1_00_000000_0
2016-05-18 00:57:54,311 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533074132_0001_1_00_000000_0
2016-05-18 00:57:54,311 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,312 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:54,312 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932] every/(1/1) ...platform/ALL_ARGS_RESULTS, VertexName: BB1DA71DD7CD40FFBDA267E54089867B, VertexParallelism: 1, TaskAttemptID:attempt_1463533074132_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=308B7803825945CA8E18257C77401370, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=AF985E64153C490DB74D308541DE72AE, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:54,312 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:54,313 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533074132_0001_1_00_000000_0
2016-05-18 00:57:54,313 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533074132_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:54,313 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533074132_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:54,315 INFO input.MRInput (MRInput.java:initialize(440)) - 308B7803825945CA8E18257C77401370 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:54,316 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 308B7803825945CA8E18257C77401370
2016-05-18 00:57:54,318 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:54,319 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533074132_0001, with dag id: dag_1463533074132_0001_1
2016-05-18 00:57:54,321 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BB1DA71DD7CD40FFBDA267E54089867B, ordinal: 0
2016-05-18 00:57:54,321 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BB1DA71DD7CD40FFBDA267E54089867B, mem on start (mb), free: 53, total: 369, max: 672
2016-05-18 00:57:54,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:54,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:54,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:54,322 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:54,323 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[308B7803825945CA8E18257C77401370:INPUT:0:org.apache.tez.mapreduce.input.MRInput], [AF985E64153C490DB74D308541DE72AE:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput]
2016-05-18 00:57:54,323 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:54,323 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:54,324 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:54,324 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[308B7803825945CA8E18257C77401370:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0], [AF985E64153C490DB74D308541DE72AE:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600]
2016-05-18 00:57:54,324 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:54,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 308B7803825945CA8E18257C77401370 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:54,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 308B7803825945CA8E18257C77401370
2016-05-18 00:57:54,325 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:54,326 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:54,326 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:54,326 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533074132_0001_1_00_000000_0
2016-05-18 00:57:54,327 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:54,327 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 308B7803825945CA8E18257C77401370 initialized RecordReader from event
2016-05-18 00:57:54,329 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:54,330 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 308B7803825945CA8E18257C77401370
2016-05-18 00:57:54,330 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 39259E4C1F20415B81AB865E7B57F7B9
2016-05-18 00:57:54,330 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 39259E4C1F20415B81AB865E7B57F7B9
2016-05-18 00:57:54,331 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:54,334 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:54,334 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - AF985E64153C490DB74D308541DE72AE using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6ac29446, comparator=cascading.tuple.hadoop.util.TupleComparator@c94b66c, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:54,334 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for AF985E64153C490DB74D308541DE72AE: , UsingHashComparator=false
2016-05-18 00:57:54,357 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:54,357 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:54,358 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for AF985E64153C490DB74D308541DE72AE: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:54,358 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - AF985E64153C490DB74D308541DE72AE: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:54,358 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:54,358 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BB1DA71DD7CD40FFBDA267E54089867B, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:54,361 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533074132_0001_1_00_000000_0
2016-05-18 00:57:54,361 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BB1DA71DD7CD40FFBDA267E54089867B, mem on close (mb), free: 163, total: 369, max: 672
2016-05-18 00:57:54,361 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - AF985E64153C490DB74D308541DE72AE: Starting flush of map output
2016-05-18 00:57:54,361 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - AF985E64153C490DB74D308541DE72AE: Span0.length = 20, perItem = 6
2016-05-18 00:57:54,362 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - AF985E64153C490DB74D308541DE72AE: done sorting span=0, length=20, time=0
2016-05-18 00:57:54,362 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - AF985E64153C490DB74D308541DE72AE: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:54,363 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - AF985E64153C490DB74D308541DE72AE: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001_wd/localmode-local-dir/output/attempt_1463533074132_0001_1_00_000000_0_10327_0/file.out
2016-05-18 00:57:54,366 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:54 Completed running task attempt: attempt_1463533074132_0001_1_00_000000_0
2016-05-18 00:57:54,366 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533074132_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:54,367 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533074132_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:54,367 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533074132_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533074359, Process_Duration=2, Process_End_Time=1463533074361, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:54,367 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533074132_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:54,368 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BB1DA71DD7CD40FFBDA267E54089867B, taskAttemptId=attempt_1463533074132_0001_1_00_000000_0, creationTime=1463533074306, allocationTime=1463533074308, startTime=1463533074310, finishTime=1463533074367, timeTaken=57, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:54,368 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:54,368 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:54,368 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,368 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533074132_0001_1_00_000000_0
2016-05-18 00:57:54,368 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:TASK_FINISHED]: vertexName=BB1DA71DD7CD40FFBDA267E54089867B, taskId=task_1463533074132_0001_1_00_000000, startTime=1463533074310, finishTime=1463533074368, timeTaken=58, status=SUCCEEDED, successfulAttemptID=attempt_1463533074132_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533074359, Process_Duration=2, Process_End_Time=1463533074361, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:54,369 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533074132_0001_00_000001
2016-05-18 00:57:54,369 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:54,369 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B]
2016-05-18 00:57:54,369 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:VERTEX_FINISHED]: vertexName=BB1DA71DD7CD40FFBDA267E54089867B, vertexId=vertex_1463533074132_0001_1_00, initRequestedTime=1463533074290, initedTime=1463533074301, startRequestedTime=1463533074291, startedTime=1463533074301, finishTime=1463533074369, timeTaken=68, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533074359, Process_Duration=2, Process_End_Time=1463533074361, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533074310, firstTasksToStart=[ task_1463533074132_0001_1_00_000000 ], lastTaskFinishTime=1463533074367, lastTasksToFinish=[ task_1463533074132_0001_1_00_000000 ], minTaskDuration=57, maxTaskDuration=57, avgTaskDuration=57.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533074132_0001_1_00_000000 ], longestDurationTasks=[ task_1463533074132_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:54,370 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:54,370 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533074132_0001_00_000001, stoppedTime=1463533074370, exitStatus=0
2016-05-18 00:57:54,370 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] attempt: attempt_1463533074132_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:54,370 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533074132_0001_1_00 [BB1DA71DD7CD40FFBDA267E54089867B] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:54,370 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:54,370 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: AF985E64153C490DB74D308541DE72AE
2016-05-18 00:57:54,370 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533074132_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:54,370 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=BB1DA71DD7CD40FFBDA267E54089867B, destinationVertexName=AF985E64153C490DB74D308541DE72AE. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:54,371 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,371 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:TASK_STARTED]: vertexName=AF985E64153C490DB74D308541DE72AE, taskId=task_1463533074132_0001_1_01_000000, scheduledTime=1463533074371, launchTime=1463533074371
2016-05-18 00:57:54,371 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,372 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533074132_0001_00_000002, launchTime=1463533074372
2016-05-18 00:57:54,373 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533074132_0001_00_000002
2016-05-18 00:57:54,373 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533074132_0001_1_01_000000_0] started. Is using containerId: [container_1463533074132_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:54,373 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AF985E64153C490DB74D308541DE72AE, taskAttemptId=attempt_1463533074132_0001_1_01_000000_0, startTime=1463533074373, containerId=container_1463533074132_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:54,374 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533074132_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533074132_0001_1_01_000000_0
2016-05-18 00:57:54 Starting to run new task attempt: attempt_1463533074132_0001_1_01_000000_0
2016-05-18 00:57:54,374 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533074132_0001_1_01_000000_0
2016-05-18 00:57:54,374 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,374 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:54,375 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932] every/(1/1) ...platform/ALL_ARGS_RESULTS, VertexName: AF985E64153C490DB74D308541DE72AE, VertexParallelism: 1, TaskAttemptID:attempt_1463533074132_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BB1DA71DD7CD40FFBDA267E54089867B, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=EEB8E00981C044D19B7579FFD891C235, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:54,375 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:54,375 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533074132_0001_1_01_000000_0
2016-05-18 00:57:54,376 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533074132_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,376 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533074132_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:54,377 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:54,379 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533074132_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:54,379 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:54,379 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533074132_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:54,380 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533074132_0001_00_000001, Exiting
2016-05-18 00:57:54,380 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533074132_0001_00_000001
2016-05-18 00:57:54,380 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533074132_0001_00_000001
2016-05-18 00:57:54,383 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533074132_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:54,383 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533074132_0001_00_000001 completed successfully
2016-05-18 00:57:54,385 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AF985E64153C490DB74D308541DE72AE, ordinal: 1
2016-05-18 00:57:54,385 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AF985E64153C490DB74D308541DE72AE, mem on start (mb), free: 158, total: 369, max: 672
2016-05-18 00:57:54,385 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:54,385 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:54,385 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:54,388 INFO output.MROutput (MROutput.java:initialize(421)) - EEB8E00981C044D19B7579FFD891C235: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:54,388 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:54,389 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[BB1DA71DD7CD40FFBDA267E54089867B:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [EEB8E00981C044D19B7579FFD891C235:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:54,389 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:54,389 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:54,389 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:54,389 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[BB1DA71DD7CD40FFBDA267E54089867B:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [EEB8E00981C044D19B7579FFD891C235:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:54,389 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:54,389 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: BB1DA71DD7CD40FFBDA267E54089867B being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:54,389 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - BB1DA71DD7CD40FFBDA267E54089867B: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:54,390 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - BB1DA71DD7CD40FFBDA267E54089867B: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:54,390 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:54,390 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:54,390 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: BB1DA71DD7CD40FFBDA267E54089867B with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:54,391 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: BB1DA71DD7CD40FFBDA267E54089867B
2016-05-18 00:57:54,391 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:54,391 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533074132_0001_1_01_000000_0
2016-05-18 00:57:54,393 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:54,394 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 39259E4C1F20415B81AB865E7B57F7B9
2016-05-18 00:57:54,394 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_RESULTS"], id: EEB8E00981C044D19B7579FFD891C235
2016-05-18 00:57:54,394 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_RESULTS"]
2016-05-18 00:57:54,394 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 39259E4C1F20415B81AB865E7B57F7B9, for 1 inputs
2016-05-18 00:57:54,395 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BB1DA71DD7CD40FFBDA267E54089867B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:54,396 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533074132_0001_1_00_000000_0_10327} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533074396, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:54,396 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : BB1DA71DD7CD40FFBDA267E54089867B
2016-05-18 00:57:54,397 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:54,397 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: BB1DA71DD7CD40FFBDA267E54089867B, wasInterrupted=false
2016-05-18 00:57:54,397 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:54,397 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:54,398 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - BB1DA71DD7CD40FFBDA267E54089867B: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:54,398 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:54,398 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001_wd/localmode-local-dir/output/attempt_1463533074132_0001_1_00_000000_0_10327/file.out, len=181, isLocal=true
2016-05-18 00:57:54,398 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:54,398 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:54,399 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:54,399 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:54,399 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : BB1DA71DD7CD40FFBDA267E54089867B
2016-05-18 00:57:54,399 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - BB1DA71DD7CD40FFBDA267E54089867B: Shuffle Runner thread complete
2016-05-18 00:57:54,399 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AF985E64153C490DB74D308541DE72AE, all 1 inputs ready in: 00:00:00.005
2016-05-18 00:57:54,400 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:54,400 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - BB1DA71DD7CD40FFBDA267E54089867B: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:54,402 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:54,404 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:54,404 INFO output.MROutput (MROutput.java:close(526)) - EEB8E00981C044D19B7579FFD891C235 closed
2016-05-18 00:57:54,404 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533074132_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:54,405 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533074132_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:54,405 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533074132_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533074399, Process_Duration=6, Process_End_Time=1463533074405, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:54,405 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:54,406 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AF985E64153C490DB74D308541DE72AE, mem on close (mb), free: 154, total: 369, max: 672
2016-05-18 00:57:54,406 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=AF985E64153C490DB74D308541DE72AE, index=1, interruptedStatus=false
2016-05-18 00:57:54,406 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: BB1DA71DD7CD40FFBDA267E54089867B
2016-05-18 00:57:54,406 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - BB1DA71DD7CD40FFBDA267E54089867B: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:54,407 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=AF985E64153C490DB74D308541DE72AE, sourceVertex=BB1DA71DD7CD40FFBDA267E54089867B, interruptedStatus=true
2016-05-18 00:57:54,407 INFO output.MROutput (MROutput.java:close(526)) - EEB8E00981C044D19B7579FFD891C235 closed
2016-05-18 00:57:54,407 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=AF985E64153C490DB74D308541DE72AE, sourceVertex=EEB8E00981C044D19B7579FFD891C235, interruptedStatus=true
2016-05-18 00:57:54,407 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533074132_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], containerShutdownRequested=false}
2016-05-18 00:57:54,407 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], containerShutdownRequested=false}
2016-05-18 00:57:54,408 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533074132_0001_00_000002
2016-05-18 00:57:54,408 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533074132_0001_00_000002
2016-05-18 00:57:54,408 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533074132_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]'}
2016-05-18 00:57:54,408 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533074132_0001_00_000002 completed but with errors
2016-05-18 00:57:54,408 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AF985E64153C490DB74D308541DE72AE, taskAttemptId=attempt_1463533074132_0001_1_01_000000_0, creationTime=1463533074371, allocationTime=1463533074372, startTime=1463533074373, finishTime=1463533074408, timeTaken=35, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533074132_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=7, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=5, LAST_EVENT_RECEIVED=5, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533074399, Process_Duration=6, Process_End_Time=1463533074405, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:54,408 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533074132_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
2016-05-18 00:57:54,408 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533074132_0001_00_000002, stoppedTime=1463533074408, exitStatus=1
2016-05-18 00:57:54,408 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533074132_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:54,408 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:TASK_FINISHED]: vertexName=AF985E64153C490DB74D308541DE72AE, taskId=task_1463533074132_0001_1_01_000000, startTime=1463533074373, finishTime=1463533074408, timeTaken=35, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533074132_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:54,409 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] because task failed: task_1463533074132_0001_1_01_000000
2016-05-18 00:57:54,409 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:54,409 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:54,409 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:54,409 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,409 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=EEB8E00981C044D19B7579FFD891C235, vertexId=vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,410 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:VERTEX_FINISHED]: vertexName=AF985E64153C490DB74D308541DE72AE, vertexId=vertex_1463533074132_0001_1_01, initRequestedTime=1463533074291, initedTime=1463533074294, startRequestedTime=1463533074301, startedTime=1463533074301, finishTime=1463533074410, timeTaken=109, status=FAILED, diagnostics=Task failed, taskId=task_1463533074132_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533074132_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533074373, firstTasksToStart=[ task_1463533074132_0001_1_01_000000 ], lastTaskFinishTime=1463533074408, lastTasksToFinish=[ task_1463533074132_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:54,410 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:54,410 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533074132_0001_1_01_000000_0 failed: true container: container_1463533074132_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:54,410 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:54,410 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:54,410 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:54,411 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533074132_0001_1, queueSize=0
2016-05-18 00:57:54,411 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074132_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533074132_0001_1, startTime=1463533074289, finishTime=1463533074410, timeTaken=121, status=FAILED, diagnostics=Vertex failed, vertexName=AF985E64153C490DB74D308541DE72AE, vertexId=vertex_1463533074132_0001_1_01, diagnostics=[Task failed, taskId=task_1463533074132_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533074132_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=210, AM_GC_TIME_MILLIS=7, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533074359, Process_Duration=2, Process_End_Time=1463533074361, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:54,411 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533074132_0001_1_01 [AF985E64153C490DB74D308541DE72AE]
2016-05-18 00:57:54,411 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533074132_0001_1 finished with state: FAILED
2016-05-18 00:57:54,411 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533074132_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:54 Completed Dag: dag_1463533074132_0001_1
2016-05-18 00:57:54,411 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533074132_0001_1, dagState=FAILED
2016-05-18 00:57:54,411 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533074132_0001_1_post
2016-05-18 00:57:54,411 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,411 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:54,411 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:54,411 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932] every/(1/1) ...platform/ALL_ARGS_RESULTS, with id=dag_1463533074132_0001_1
2016-05-18 00:57:54,412 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932] every/(1/1) ...platform/ALL_ARGS_RESULTS, with id=dag_1463533074132_0001_1
2016-05-18 00:57:54,421 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...platform/ALL_ARGS_RESULTS, applicationId=application_1463533074132_0001
2016-05-18 00:57:54,422 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:54,422 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:54,422 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:54,422 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:54,422 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:54,423 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:54,423 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:54,423 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:54,423 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:54,423 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:54,423 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:54,424 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:54,424 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:54,425 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:54,428 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/D4A3C8D1D2D544AF810EA3ED310D239C/CC17D4FF2238436E9B8D94474BE7F932/.tez/application_1463533074132_0001
2016-05-18 00:57:54,428 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:54,429 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...platform/ALL_ARGS_RESULTS
2016-05-18 00:57:54,429 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:54,429 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_RESULTS/_temporary
2016-05-18 00:57:54,429 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.317
2016-05-18 00:57:54,430 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_ARGS_RESULTS
2016-05-18 00:57:54,454 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:54,455 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:54,481 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.002
2016-05-18 00:57:54,482 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.003
2016-05-18 00:57:54,482 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:54,482 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-18 00:57:54,483 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_ARGS_ALL
2016-05-18 00:57:54,503 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:54,504 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:54,532 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.006
2016-05-18 00:57:54,538 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.011
2016-05-18 00:57:54,539 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:54,539 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:54,539 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:54,554 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:54,558 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_REPLACE
2016-05-18 00:57:54,602 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:54,602 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:54,602 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_REPLACE"]
2016-05-18 00:57:54,602 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:54,602 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:54,602 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:54,602 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...platform/ALL_ARGS_REPLACE
2016-05-18 00:57:54,602 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:54,610 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:54,610 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:54,617 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:54,621 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:54,621 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:54,622 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:54,631 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001 doesn't exist and is created
2016-05-18 00:57:54,645 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:54,645 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:54,645 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001_wd
2016-05-18 00:57:54,655 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533074621_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:54,655 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:54,655 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:54,656 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:54,656 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:54,656 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:54,656 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:54,657 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:54,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:54,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:54,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:54,658 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:54,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:54,659 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:54,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:54,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:54,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:54,659 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:54,660 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:54,660 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:54,661 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:54,662 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:54,662 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:54,662 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001_wd/localmode-log-dir
2016-05-18 00:57:54,662 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001_wd/localmode-log-dir/history.txt.appattempt_1463533074621_0001_000000, maxErrors=10
2016-05-18 00:57:54,663 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:54,663 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533074621_0001_000000, appSubmitTime=1463533074654, launchTime=1463533074654
2016-05-18 00:57:54,668 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:54,668 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:37313
2016-05-18 00:57:54,668 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533074621
2016-05-18 00:57:54,671 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533074621_0001_000000, startTime=1463533074671
2016-05-18 00:57:54,671 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:54,745 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:54,745 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:54,745 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...platform/ALL_ARGS_REPLACE, applicationId=application_1463533074621_0001, dagName=[87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0] every/(1/1) ...platform/ALL_ARGS_REPLACE
2016-05-18 00:57:54,759 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0] every/(1/1) ...platform/ALL_ARGS_REPLACE
2016-05-18 00:57:54,760 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:54,760 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533074621_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001_wd/localmode-log-dir/dag_1463533074621_0001_1.dot
2016-05-18 00:57:54,761 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001_wd/localmode-log-dir/dag_1463533074621_0001_1-tez-dag.pb.txt
2016-05-18 00:57:54,766 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533074621_0001_1
2016-05-18 00:57:54,766 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,766 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0] every/(1/1) ...platform/ALL_ARGS_REPLACE
2016-05-18 00:57:54 Running Dag: dag_1463533074621_0001_1
2016-05-18 00:57:54,776 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533074621_0001_1, submitTime=1463533074760
2016-05-18 00:57:54,776 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001/tez-conf.pb" } size: 36913 timestamp: 1463533074000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:54,777 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,777 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:54,777 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001/tez-conf.pb" } size: 36913 timestamp: 1463533074000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:54,777 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,777 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:54,778 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:54,778 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533074621_0001_1, initTime=1463533074778
2016-05-18 00:57:54,778 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533074621_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:54,778 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:54,778 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:DAG_STARTED]: dagID=dag_1463533074621_0001_1, startTime=1463533074778
2016-05-18 00:57:54,778 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533074621_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:54,778 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: AB6CB933A8254328BF0FB9304EB3CC67 : {63EC33C6838140E6884C2E7F4B8D9F76={InputName=63EC33C6838140E6884C2E7F4B8D9F76}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:54,779 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: 63EC33C6838140E6884C2E7F4B8D9F76, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:54,779 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,779 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,779 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,779 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,780 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...platform/ALL_ARGS_REPLACE, applicationId=application_1463533074621_0001, dagName=[87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0] every/(1/1) ...platform/ALL_ARGS_REPLACE
2016-05-18 00:57:54,780 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:54,780 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,781 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: 63EC33C6838140E6884C2E7F4B8D9F76 on vertex vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,783 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input 63EC33C6838140E6884C2E7F4B8D9F76 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:54,783 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:54,783 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,783 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,784 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533074621_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:54,784 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4], numAdditionalOutputs=1
2016-05-18 00:57:54,784 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=63BC764CE28840FC83C4543CDEEE9EED, vertex=vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:54,785 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:54,785 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:54,786 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:54,786 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:54,787 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A0C466749DE04477918EBE20E32668E4:63BC764CE28840FC83C4543CDEEE9EED using oldmapred API
2016-05-18 00:57:54,787 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=63BC764CE28840FC83C4543CDEEE9EED, vertexName=A0C466749DE04477918EBE20E32668E4, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:54,787 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: 63EC33C6838140E6884C2E7F4B8D9F76 on vertex vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,788 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A0C466749DE04477918EBE20E32668E4, vertexId=vertex_1463533074621_0001_1_01, initRequestedTime=1463533074780, initedTime=1463533074784, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:54,788 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:54,789 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:54,789 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {63EC33C6838140E6884C2E7F4B8D9F76=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:54,789 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] parallelism set to 1
2016-05-18 00:57:54,789 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:54,790 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AB6CB933A8254328BF0FB9304EB3CC67, vertexId=vertex_1463533074621_0001_1_00, initRequestedTime=1463533074778, initedTime=1463533074790, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:54,790 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] with distanceFromRoot: 0
2016-05-18 00:57:54,790 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:54,790 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533074621_0001_1_00, startRequestedTime=1463533074780, startedTime=1463533074790
2016-05-18 00:57:54,790 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533074621_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:54,790 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:54,790 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533074621_0001_1_00 for vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] numStartedSources: 1 numSources: 1
2016-05-18 00:57:54,790 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] with distanceFromRoot: 1
2016-05-18 00:57:54,791 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533074621_0001_1_01, startRequestedTime=1463533074790, startedTime=1463533074790
2016-05-18 00:57:54,791 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:54,792 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A0C466749DE04477918EBE20E32668E4 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:54,792 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: AB6CB933A8254328BF0FB9304EB3CC67 in vertex: A0C466749DE04477918EBE20E32668E4 numBipartiteSourceTasks: 1
2016-05-18 00:57:54,793 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in AB6CB933A8254328BF0FB9304EB3CC67
2016-05-18 00:57:54,793 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,793 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:TASK_STARTED]: vertexName=AB6CB933A8254328BF0FB9304EB3CC67, taskId=task_1463533074621_0001_1_00_000000, scheduledTime=1463533074793, launchTime=1463533074793
2016-05-18 00:57:54,794 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,794 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:54,796 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533074621_0001_00_000001, launchTime=1463533074796
2016-05-18 00:57:54,796 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533074621_0001_00_000001
2016-05-18 00:57:54,797 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533074621_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533074621_0001_1_00_000000_0
2016-05-18 00:57:54,797 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533074621_0001_1_00_000000_0] started. Is using containerId: [container_1463533074621_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:54,797 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AB6CB933A8254328BF0FB9304EB3CC67, taskAttemptId=attempt_1463533074621_0001_1_00_000000_0, startTime=1463533074797, containerId=container_1463533074621_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:54 Starting to run new task attempt: attempt_1463533074621_0001_1_00_000000_0
2016-05-18 00:57:54,797 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533074621_0001_1_00_000000_0
2016-05-18 00:57:54,797 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,797 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:54,797 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0] every/(1/1) ...platform/ALL_ARGS_REPLACE, VertexName: AB6CB933A8254328BF0FB9304EB3CC67, VertexParallelism: 1, TaskAttemptID:attempt_1463533074621_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=63EC33C6838140E6884C2E7F4B8D9F76, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A0C466749DE04477918EBE20E32668E4, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:54,798 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:54,798 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533074621_0001_1_00_000000_0
2016-05-18 00:57:54,798 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533074621_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:54,798 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533074621_0001, with dag id: dag_1463533074621_0001_1
2016-05-18 00:57:54,800 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533074621_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:54,801 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:54,804 INFO input.MRInput (MRInput.java:initialize(440)) - 63EC33C6838140E6884C2E7F4B8D9F76 using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:54,804 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: 63EC33C6838140E6884C2E7F4B8D9F76
2016-05-18 00:57:54,805 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AB6CB933A8254328BF0FB9304EB3CC67, ordinal: 0
2016-05-18 00:57:54,805 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AB6CB933A8254328BF0FB9304EB3CC67, mem on start (mb), free: 51, total: 369, max: 672
2016-05-18 00:57:54,806 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:54,806 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:54,806 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:54,806 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:54,806 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A0C466749DE04477918EBE20E32668E4:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [63EC33C6838140E6884C2E7F4B8D9F76:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 00:57:54,806 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:54,807 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:54,807 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:54,807 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A0C466749DE04477918EBE20E32668E4:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [63EC33C6838140E6884C2E7F4B8D9F76:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 00:57:54,807 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:54,807 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: 63EC33C6838140E6884C2E7F4B8D9F76 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:54,807 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: 63EC33C6838140E6884C2E7F4B8D9F76
2016-05-18 00:57:54,807 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:54,808 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:54,808 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:54,808 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533074621_0001_1_00_000000_0
2016-05-18 00:57:54,809 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:54,809 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - 63EC33C6838140E6884C2E7F4B8D9F76 initialized RecordReader from event
2016-05-18 00:57:54,811 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:54,812 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: 63EC33C6838140E6884C2E7F4B8D9F76
2016-05-18 00:57:54,812 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: C7B9D5F6D6C94136B9CD6E3CB397F28E
2016-05-18 00:57:54,812 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] C7B9D5F6D6C94136B9CD6E3CB397F28E
2016-05-18 00:57:54,813 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:54,816 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:54,816 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A0C466749DE04477918EBE20E32668E4 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@6ba9ad80, comparator=cascading.tuple.hadoop.util.TupleComparator@6334bc71, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:54,817 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A0C466749DE04477918EBE20E32668E4: , UsingHashComparator=false
2016-05-18 00:57:54,839 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:54,839 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:54,839 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A0C466749DE04477918EBE20E32668E4: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:54,840 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A0C466749DE04477918EBE20E32668E4: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:54,840 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:54,840 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AB6CB933A8254328BF0FB9304EB3CC67, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:54,843 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533074621_0001_1_00_000000_0
2016-05-18 00:57:54,844 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AB6CB933A8254328BF0FB9304EB3CC67, mem on close (mb), free: 161, total: 369, max: 672
2016-05-18 00:57:54,844 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A0C466749DE04477918EBE20E32668E4: Starting flush of map output
2016-05-18 00:57:54,844 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A0C466749DE04477918EBE20E32668E4: Span0.length = 20, perItem = 6
2016-05-18 00:57:54,844 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A0C466749DE04477918EBE20E32668E4: done sorting span=0, length=20, time=0
2016-05-18 00:57:54,844 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A0C466749DE04477918EBE20E32668E4: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:54,846 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A0C466749DE04477918EBE20E32668E4: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001_wd/localmode-local-dir/output/attempt_1463533074621_0001_1_00_000000_0_10332_0/file.out
2016-05-18 00:57:54,851 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:54 Completed running task attempt: attempt_1463533074621_0001_1_00_000000_0
2016-05-18 00:57:54,851 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533074621_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:54,851 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533074621_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:54,851 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533074621_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533074840, Process_Duration=3, Process_End_Time=1463533074843, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:54,852 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533074621_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:54,852 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:54,852 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AB6CB933A8254328BF0FB9304EB3CC67, taskAttemptId=attempt_1463533074621_0001_1_00_000000_0, creationTime=1463533074793, allocationTime=1463533074795, startTime=1463533074797, finishTime=1463533074852, timeTaken=55, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:54,852 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533074621_0001_1_00_000000_0
2016-05-18 00:57:54,853 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:TASK_FINISHED]: vertexName=AB6CB933A8254328BF0FB9304EB3CC67, taskId=task_1463533074621_0001_1_00_000000, startTime=1463533074797, finishTime=1463533074852, timeTaken=55, status=SUCCEEDED, successfulAttemptID=attempt_1463533074621_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533074840, Process_Duration=3, Process_End_Time=1463533074843, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:54,853 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:54,853 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67]
2016-05-18 00:57:54,852 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:54,853 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,854 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:VERTEX_FINISHED]: vertexName=AB6CB933A8254328BF0FB9304EB3CC67, vertexId=vertex_1463533074621_0001_1_00, initRequestedTime=1463533074778, initedTime=1463533074790, startRequestedTime=1463533074780, startedTime=1463533074790, finishTime=1463533074853, timeTaken=63, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533074840, Process_Duration=3, Process_End_Time=1463533074843, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533074797, firstTasksToStart=[ task_1463533074621_0001_1_00_000000 ], lastTaskFinishTime=1463533074852, lastTasksToFinish=[ task_1463533074621_0001_1_00_000000 ], minTaskDuration=55, maxTaskDuration=55, avgTaskDuration=55.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533074621_0001_1_00_000000 ], longestDurationTasks=[ task_1463533074621_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:54,854 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:54,854 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533074621_0001_00_000001, stoppedTime=1463533074854, exitStatus=0
2016-05-18 00:57:54,854 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] attempt: attempt_1463533074621_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:54,854 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533074621_0001_1_00 [AB6CB933A8254328BF0FB9304EB3CC67] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:54,854 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:54,854 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533074621_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:54,855 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A0C466749DE04477918EBE20E32668E4
2016-05-18 00:57:54,855 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=AB6CB933A8254328BF0FB9304EB3CC67, destinationVertexName=A0C466749DE04477918EBE20E32668E4. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:54,855 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,855 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533074621_0001_00_000001
2016-05-18 00:57:54,855 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:TASK_STARTED]: vertexName=A0C466749DE04477918EBE20E32668E4, taskId=task_1463533074621_0001_1_01_000000, scheduledTime=1463533074855, launchTime=1463533074855
2016-05-18 00:57:54,856 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,857 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533074621_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:54,857 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533074621_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:54,857 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533074621_0001_00_000001, Exiting
2016-05-18 00:57:54,857 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533074621_0001_00_000001
2016-05-18 00:57:54,857 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533074621_0001_00_000001
2016-05-18 00:57:54,858 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533074621_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:54,858 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533074621_0001_00_000001 completed successfully
2016-05-18 00:57:54,858 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533074621_0001_00_000002, launchTime=1463533074857
2016-05-18 00:57:54,858 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533074621_0001_00_000002
2016-05-18 00:57:54,859 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533074621_0001_1_01_000000_0] started. Is using containerId: [container_1463533074621_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:54,859 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A0C466749DE04477918EBE20E32668E4, taskAttemptId=attempt_1463533074621_0001_1_01_000000_0, startTime=1463533074859, containerId=container_1463533074621_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:54,859 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533074621_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533074621_0001_1_01_000000_0
2016-05-18 00:57:54 Starting to run new task attempt: attempt_1463533074621_0001_1_01_000000_0
2016-05-18 00:57:54,860 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533074621_0001_1_01_000000_0
2016-05-18 00:57:54,860 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,860 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:54,861 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0] every/(1/1) ...platform/ALL_ARGS_REPLACE, VertexName: A0C466749DE04477918EBE20E32668E4, VertexParallelism: 1, TaskAttemptID:attempt_1463533074621_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AB6CB933A8254328BF0FB9304EB3CC67, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=63BC764CE28840FC83C4543CDEEE9EED, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:54,861 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:54,861 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533074621_0001_1_01_000000_0
2016-05-18 00:57:54,862 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533074621_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,862 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533074621_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:54,864 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:54,867 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:54,871 INFO output.MROutput (MROutput.java:initialize(421)) - 63BC764CE28840FC83C4543CDEEE9EED: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:54,872 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A0C466749DE04477918EBE20E32668E4, ordinal: 1
2016-05-18 00:57:54,872 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A0C466749DE04477918EBE20E32668E4, mem on start (mb), free: 156, total: 369, max: 672
2016-05-18 00:57:54,872 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:54,872 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:54,872 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:54,872 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:54,872 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[AB6CB933A8254328BF0FB9304EB3CC67:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput], [63BC764CE28840FC83C4543CDEEE9EED:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput]
2016-05-18 00:57:54,872 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:54,873 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:54,873 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:54,873 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[AB6CB933A8254328BF0FB9304EB3CC67:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912], [63BC764CE28840FC83C4543CDEEE9EED:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0]
2016-05-18 00:57:54,873 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:54,874 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: AB6CB933A8254328BF0FB9304EB3CC67 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:54,874 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:54,875 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:54,874 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - AB6CB933A8254328BF0FB9304EB3CC67: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:54,875 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - AB6CB933A8254328BF0FB9304EB3CC67: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:54,876 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: AB6CB933A8254328BF0FB9304EB3CC67 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:54,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: AB6CB933A8254328BF0FB9304EB3CC67
2016-05-18 00:57:54,877 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:54,877 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533074621_0001_1_01_000000_0
2016-05-18 00:57:54,878 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AB6CB933A8254328BF0FB9304EB3CC67: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:54,878 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533074621_0001_1_00_000000_0_10332} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533074878, TimeTaken=0, Rate=0.00 MB/s
2016-05-18 00:57:54,878 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : AB6CB933A8254328BF0FB9304EB3CC67
2016-05-18 00:57:54,879 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:54,879 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: AB6CB933A8254328BF0FB9304EB3CC67, wasInterrupted=false
2016-05-18 00:57:54,879 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:54,879 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:54,880 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:54,880 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001_wd/localmode-local-dir/output/attempt_1463533074621_0001_1_00_000000_0_10332/file.out, len=181, isLocal=true
2016-05-18 00:57:54,880 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:54,880 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:54,880 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:54,881 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:54,881 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : AB6CB933A8254328BF0FB9304EB3CC67
2016-05-18 00:57:54,881 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - AB6CB933A8254328BF0FB9304EB3CC67: Shuffle Runner thread complete
2016-05-18 00:57:54,881 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:54,883 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: C7B9D5F6D6C94136B9CD6E3CB397F28E
2016-05-18 00:57:54,883 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_REPLACE"], id: 63BC764CE28840FC83C4543CDEEE9EED
2016-05-18 00:57:54,883 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_REPLACE"]
2016-05-18 00:57:54,883 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] C7B9D5F6D6C94136B9CD6E3CB397F28E, for 1 inputs
2016-05-18 00:57:54,884 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A0C466749DE04477918EBE20E32668E4, all 1 inputs ready in: 00:00:00.001
2016-05-18 00:57:54,884 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:54,884 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - AB6CB933A8254328BF0FB9304EB3CC67: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:54,886 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:54,888 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:54,889 INFO output.MROutput (MROutput.java:close(526)) - 63BC764CE28840FC83C4543CDEEE9EED closed
2016-05-18 00:57:54,889 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533074621_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:54,889 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533074621_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:54,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533074621_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=6, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533074884, Process_Duration=5, Process_End_Time=1463533074889, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:54,890 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:54,890 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A0C466749DE04477918EBE20E32668E4, mem on close (mb), free: 153, total: 369, max: 672
2016-05-18 00:57:54,891 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=A0C466749DE04477918EBE20E32668E4, index=1, interruptedStatus=false
2016-05-18 00:57:54,891 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: AB6CB933A8254328BF0FB9304EB3CC67
2016-05-18 00:57:54,891 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - AB6CB933A8254328BF0FB9304EB3CC67: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:54,891 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=A0C466749DE04477918EBE20E32668E4, sourceVertex=AB6CB933A8254328BF0FB9304EB3CC67, interruptedStatus=true
2016-05-18 00:57:54,891 INFO output.MROutput (MROutput.java:close(526)) - 63BC764CE28840FC83C4543CDEEE9EED closed
2016-05-18 00:57:54,891 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=A0C466749DE04477918EBE20E32668E4, sourceVertex=63BC764CE28840FC83C4543CDEEE9EED, interruptedStatus=true
2016-05-18 00:57:54,892 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533074621_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], containerShutdownRequested=false}
2016-05-18 00:57:54,892 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], containerShutdownRequested=false}
2016-05-18 00:57:54,892 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533074621_0001_00_000002
2016-05-18 00:57:54,892 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533074621_0001_00_000002
2016-05-18 00:57:54,893 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533074621_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]'}
2016-05-18 00:57:54,892 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A0C466749DE04477918EBE20E32668E4, taskAttemptId=attempt_1463533074621_0001_1_01_000000_0, creationTime=1463533074856, allocationTime=1463533074856, startTime=1463533074859, finishTime=1463533074892, timeTaken=33, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533074621_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=4, MERGE_PHASE_TIME=6, FIRST_EVENT_RECEIVED=3, LAST_EVENT_RECEIVED=3, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533074884, Process_Duration=5, Process_End_Time=1463533074889, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:54,893 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533074621_0001_00_000002 completed but with errors
2016-05-18 00:57:54,893 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533074621_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:54,893 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:TASK_FINISHED]: vertexName=A0C466749DE04477918EBE20E32668E4, taskId=task_1463533074621_0001_1_01_000000, startTime=1463533074859, finishTime=1463533074893, timeTaken=34, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533074621_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:54,893 INFO container.AMContainerImpl (AMContainerImpl.java:transition(704)) - Container container_1463533074621_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
2016-05-18 00:57:54,894 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533074621_0001_00_000002, stoppedTime=1463533074893, exitStatus=1
2016-05-18 00:57:54,894 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] because task failed: task_1463533074621_0001_1_01_000000
2016-05-18 00:57:54,894 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:54,894 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:54,894 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:54,894 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,894 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=63BC764CE28840FC83C4543CDEEE9EED, vertexId=vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,895 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:VERTEX_FINISHED]: vertexName=A0C466749DE04477918EBE20E32668E4, vertexId=vertex_1463533074621_0001_1_01, initRequestedTime=1463533074780, initedTime=1463533074784, startRequestedTime=1463533074790, startedTime=1463533074790, finishTime=1463533074895, timeTaken=105, status=FAILED, diagnostics=Task failed, taskId=task_1463533074621_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533074621_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533074859, firstTasksToStart=[ task_1463533074621_0001_1_01_000000 ], lastTaskFinishTime=1463533074892, lastTasksToFinish=[ task_1463533074621_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:54,896 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:54,896 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533074621_0001_1_01_000000_0 failed: true container: container_1463533074621_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:54,896 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:54,896 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:54,896 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:54,897 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533074621_0001_1, queueSize=0
2016-05-18 00:57:54,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533074621_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533074621_0001_1, startTime=1463533074778, finishTime=1463533074896, timeTaken=118, status=FAILED, diagnostics=Vertex failed, vertexName=A0C466749DE04477918EBE20E32668E4, vertexId=vertex_1463533074621_0001_1_01, diagnostics=[Task failed, taskId=task_1463533074621_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533074621_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=190, AM_GC_TIME_MILLIS=8, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533074840, Process_Duration=3, Process_End_Time=1463533074843, Read_Duration=1, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:54,897 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533074621_0001_1_01 [A0C466749DE04477918EBE20E32668E4]
2016-05-18 00:57:54,897 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533074621_0001_1 finished with state: FAILED
2016-05-18 00:57:54,897 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533074621_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:54 Completed Dag: dag_1463533074621_0001_1
2016-05-18 00:57:54,898 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533074621_0001_1, dagState=FAILED
2016-05-18 00:57:54,898 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533074621_0001_1_post
2016-05-18 00:57:54,898 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:54,898 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-05-18 00:57:54,898 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:54,898 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0] every/(1/1) ...platform/ALL_ARGS_REPLACE, with id=dag_1463533074621_0001_1
2016-05-18 00:57:54,898 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0] every/(1/1) ...platform/ALL_ARGS_REPLACE, with id=dag_1463533074621_0001_1
2016-05-18 00:57:54,906 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...platform/ALL_ARGS_REPLACE, applicationId=application_1463533074621_0001
2016-05-18 00:57:54,907 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:54,907 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:54,908 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:54,908 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:54,908 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:54,908 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:54,908 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:54,908 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:54,908 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:54,908 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:54,909 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:54,909 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:54,909 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:54,910 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:54,913 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/87CA7B95E5F94563A1D4E60665127959/ADA944C5EE074594BA7B5E6FD461D3C0/.tez/application_1463533074621_0001
2016-05-18 00:57:54,913 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:54,914 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...platform/ALL_ARGS_REPLACE
2016-05-18 00:57:54,914 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:54,914 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_REPLACE/_temporary
2016-05-18 00:57:54,915 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.312
2016-05-18 00:57:54,915 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_ARGS_REPLACE
2016-05-18 00:57:54,936 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-18 00:57:54,936 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-18 00:57:54,964 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.004
2016-05-18 00:57:54,969 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.009
2016-05-18 00:57:54,969 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:54,970 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-18 00:57:54,970 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-18 00:57:54,982 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt
2016-05-18 00:57:54,987 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_SWAP
2016-05-18 00:57:55,029 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting
2016-05-18 00:57:55,029 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:55,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] sink: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_SWAP"]
2016-05-18 00:57:55,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] parallel execution of steps is enabled: false
2016-05-18 00:57:55,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] executing total steps: 1
2016-05-18 00:57:55,030 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] allocating management threads: 1
2016-05-18 00:57:55,031 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] starting step: (1/1) ...blyplatform/ALL_ARGS_SWAP
2016-05-18 00:57:55,031 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [every] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-18 00:57:55,042 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] tez session mode enabled: true
2016-05-18 00:57:55,047 INFO client.TezClient (TezClient.java:<init>(174)) - Tez Client Version: [ component=tez-api, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:55,047 INFO client.TezClient (TezClient.java:start(394)) - Session mode. Starting session.
2016-05-18 00:57:55,047 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(164)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-18 00:57:55,056 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(122)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001 doesn't exist and is created
2016-05-18 00:57:55,070 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(332)) - DAGAppMaster thread has been created
2016-05-18 00:57:55,070 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-18 00:57:55,070 INFO client.LocalClient (LocalClient.java:run(282)) - Using working directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001_wd
2016-05-18 00:57:55,079 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(363)) - Created DAGAppMaster for application appattempt_1463533075047_0001_000000, versionInfo=[ component=tez-dag, version=0.8.2, revision=2d46dcefa492a6957cfa972007039f7e188760ce, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160112-1834 ]
2016-05-18 00:57:55,080 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(80)) - Trying to locate HadoopShimProvider for hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:55,080 INFO shim.HadoopShimsLoader (HadoopShimsLoader.java:<init>(139)) - Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=2.6.4, majorVersion=2, minorVersion=6
2016-05-18 00:57:55,081 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(436)) - AM Level configured TaskSchedulers: [0:TezUber:null]
2016-05-18 00:57:55,081 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(437)) - AM Level configured ContainerLaunchers: [0:TezUber:null]
2016-05-18 00:57:55,081 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(438)) - AM Level configured TaskCommunicators: [0:TezUber:null]
2016-05-18 00:57:55,082 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(447)) - Comparing client version with AM version, clientVersion=0.8.2, AMVersion=0.8.2
2016-05-18 00:57:55,082 INFO app.TaskCommunicatorManager (TaskCommunicatorManager.java:createUberTaskCommunicator(192)) - Creating Default Local Task Communicator
2016-05-18 00:57:55,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.container.AMContainerEventType for class org.apache.tez.dag.app.rm.container.AMContainerMap
2016-05-18 00:57:55,083 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.node.AMNodeEventType for class org.apache.tez.dag.app.rm.node.AMNodeTracker
2016-05-18 00:57:55,084 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGAppMasterEventType for class org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterEventHandler
2016-05-18 00:57:55,084 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.DAGEventType for class org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher
2016-05-18 00:57:55,084 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.VertexEventType for class org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher
2016-05-18 00:57:55,084 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(529)) - Using concurrent dispatcher: false
2016-05-18 00:57:55,084 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskEventDispatcher
2016-05-18 00:57:55,084 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.TaskAttemptEventType for class org.apache.tez.dag.app.DAGAppMaster$TaskAttemptEventDispatcher
2016-05-18 00:57:55,085 INFO common.AsyncDispatcher (AsyncDispatcher.java:registerAndCreateDispatcher(262)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for independent dispatch using: class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:55,085 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.dag.event.SpeculatorEventType for class org.apache.tez.dag.app.DAGAppMaster$SpeculatorEventHandler
2016-05-18 00:57:55,085 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.AMSchedulerEventType for class org.apache.tez.dag.app.rm.TaskSchedulerManager
2016-05-18 00:57:55,085 INFO launcher.TezContainerLauncherImpl (ContainerLauncherManager.java:createUberContainerLauncher(134)) - Creating LocalContainerLauncher
2016-05-18 00:57:55,087 INFO common.AsyncDispatcher (AsyncDispatcher.java:register(236)) - Registering class org.apache.tez.dag.app.rm.ContainerLauncherEventType for class org.apache.tez.dag.app.launcher.ContainerLauncherManager
2016-05-18 00:57:55,087 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(78)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1, nodeUpdatesRescheduleEnabled: false
2016-05-18 00:57:55,087 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(57)) - Initializing HistoryEventHandler withrecoveryEnabled=true, historyServiceClassName=org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
2016-05-18 00:57:55,088 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001_wd/localmode-log-dir
2016-05-18 00:57:55,088 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001_wd/localmode-log-dir/history.txt.appattempt_1463533075047_0001_000000, maxErrors=10
2016-05-18 00:57:55,088 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(124)) - RecoveryService initialized with recoveryPath=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001/recovery/0, bufferSize(bytes)=8192, flushInterval(s)=30, maxUnflushedEvents=100
2016-05-18 00:57:55,089 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463533075047_0001_000000, appSubmitTime=1463533075079, launchTime=1463533075079
2016-05-18 00:57:55,092 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-18 00:57:55,094 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:57909
2016-05-18 00:57:55,094 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:createUberTaskScheduler(557)) - Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=1463533075047
2016-05-18 00:57:55,096 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463533075047_0001_000000, startTime=1463533075096
2016-05-18 00:57:55,096 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1945)) - In Session mode. Waiting for DAG over RPC
2016-05-18 00:57:55,114 INFO app.DAGAppMaster (DAGAppMaster.java:run(946)) - Calling stop for all the services
2016-05-18 00:57:55,114 INFO app.DAGAppMaster (DAGAppMaster.java:run(957)) - Exiting DAGAppMaster..GoodBye!
2016-05-18 00:57:55,170 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-18 00:57:55,170 INFO client.TezClient (TezClient.java:start(428)) - The url to track the Tez Session: N/A
2016-05-18 00:57:55,170 INFO client.TezClient (TezClient.java:submitDAGSession(470)) - Submitting dag to TezSession, sessionName=(1/1) ...blyplatform/ALL_ARGS_SWAP, applicationId=application_1463533075047_0001, dagName=[E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6] every/(1/1) ...blyplatform/ALL_ARGS_SWAP
2016-05-18 00:57:55,184 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1351)) - Starting DAG submitted via RPC: [E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6] every/(1/1) ...blyplatform/ALL_ARGS_SWAP
2016-05-18 00:57:55,185 INFO app.DAGAppMaster (DAGAppMaster.java:createDAG(985)) - Merging AM credentials into DAG credentials
2016-05-18 00:57:55,185 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(1076)) - Generating DAG graphviz file, dagId=dag_1463533075047_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001_wd/localmode-log-dir/dag_1463533075047_0001_1.dot
2016-05-18 00:57:55,186 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(1093)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001_wd/localmode-log-dir/dag_1463533075047_0001_1-tez-dag.pb.txt
2016-05-18 00:57:55,193 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533075047_0001_1
2016-05-18 00:57:55,193 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:55,193 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2399)) - Running DAG: [E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6] every/(1/1) ...blyplatform/ALL_ARGS_SWAP
2016-05-18 00:57:55 Running Dag: dag_1463533075047_0001_1
2016-05-18 00:57:55,202 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463533075047_0001_1, submitTime=1463533075185
2016-05-18 00:57:55,202 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001/tez-conf.pb" } size: 36913 timestamp: 1463533075000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:55,203 INFO impl.VertexImpl (VertexImpl.java:setAdditionalInputs(3978)) - Setting 1 additional inputs for vertexvertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,203 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:55,204 INFO impl.VertexImpl (VertexImpl.java:<init>(928)) - Default container context for vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]=LocalResources: [[ name=tez-conf.pb, value=resource { scheme: "file" port: -1 file: "/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001/tez-conf.pb" } size: 36913 timestamp: 1463533075000 type: FILE visibility: APPLICATION],], environment: [[ CLASSPATH=$PWD/cascading-hadoop2-tez-3.1.0.jar/:$PWD/cascading-hadoop2-tez-3.1.0.jar/classes/:$PWD/cascading-hadoop2-tez-3.1.0.jar/lib/*::$PWD:$PWD/*:$HADOOP_CONF_DIR: ],[ LD_LIBRARY_PATH=$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native/ ],], credentials(token kinds): [tez.job,], javaOpts: -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA, vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14], Default Resources=<memory:1024, vCores:1>
2016-05-18 00:57:55,204 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4023)) - Setting 1 additional outputs for vertex vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,205 INFO impl.VertexImpl (VertexImpl.java:<init>(1016)) - Running vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] : TaskScheduler=0:TezUber, ContainerLauncher=0:TezUber, TaskCommunicator=0:TezUber
2016-05-18 00:57:55,205 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1593)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-18 00:57:55,205 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463533075047_0001_1, initTime=1463533075205
2016-05-18 00:57:55,205 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533075047_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-18 00:57:55,205 INFO impl.DAGImpl (DAGImpl.java:transition(1768)) - Added additional resources : [[]] to classpath
2016-05-18 00:57:55,205 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:DAG_STARTED]: dagID=dag_1463533075047_0001_1, startTime=1463533075205
2016-05-18 00:57:55,205 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533075047_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-18 00:57:55,206 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2400)) - Root Inputs exist for Vertex: A90BB0BD5F2640B6B3D0FC04C551A31C : {ED4172E4561A4AE5933B8AFD35FA9D0F={InputName=ED4172E4561A4AE5933B8AFD35FA9D0F}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-18 00:57:55,206 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2410)) - Starting root input initializer for input: ED4172E4561A4AE5933B8AFD35FA9D0F, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-18 00:57:55,206 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2550)) - Setting vertexManager to RootInputVertexManager for vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,206 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2721)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,206 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2726)) - Vertex will initialize from input initializer. vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,206 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3893)) - Starting 1 inputInitializers for vertex vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,207 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-18 00:57:55,207 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2562)) - Setting vertexManager to ShuffleVertexManager for vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,207 INFO client.TezClient (TezClient.java:submitDAGSession(545)) - Submitted dag to TezSession, sessionName=(1/1) ...blyplatform/ALL_ARGS_SWAP, applicationId=application_1463533075047_0001, dagName=[E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6] every/(1/1) ...blyplatform/ALL_ARGS_SWAP
2016-05-18 00:57:55,209 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(272)) - Starting InputInitializer for Input: ED4172E4561A4AE5933B8AFD35FA9D0F on vertex vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,209 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(1022)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-18 00:57:55,209 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2760)) - Creating 1 tasks for vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,210 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2779)) - Directly initializing vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,210 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533075047_0001_1_01, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0, setParallelismCalledFlag=false
2016-05-18 00:57:55,210 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2187)) - Setting up committers for vertex vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14], numAdditionalOutputs=1
2016-05-18 00:57:55,210 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(2202)) - Instantiating committer for output=743B00DF4205494EB060C46586059969, vertex=vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-18 00:57:55,212 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(109)) - Input ED4172E4561A4AE5933B8AFD35FA9D0F asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7, groupingEnabled: true
2016-05-18 00:57:55,212 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(121)) - Committer for A3D788584351473E8D00CD9B27867E14:743B00DF4205494EB060C46586059969 using oldmapred API
2016-05-18 00:57:55,213 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(146)) - OutputCommitter for outputName=743B00DF4205494EB060C46586059969, vertexName=A3D788584351473E8D00CD9B27867E14, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-18 00:57:55,213 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A3D788584351473E8D00CD9B27867E14, vertexId=vertex_1463533075047_0001_1_01, initRequestedTime=1463533075207, initedTime=1463533075210, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0, initGeneratedEventsCount=0
2016-05-18 00:57:55,214 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] transitioned from NEW to INITED due to event V_INIT
2016-05-18 00:57:55,214 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(168)) - Grouping splits in Tez
2016-05-18 00:57:55,214 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(226)) - Desired splits: 1 too large. Desired splitLength: 51 Min splitLength: 52428800 New desired splits: 1 Total length: 51 Original splits: 2
2016-05-18 00:57:55,214 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(321)) - Desired numSplits: 1 lengthPerGroup: 51 numLocations: 1 numSplitsPerLocation: 2 numSplitsInGroup: 2 totalLength: 51 numOriginalSplits: 2 . Grouping by length: true count: false nodeLocalOnly: false
2016-05-18 00:57:55,214 INFO grouper.TezSplitGrouper (TezSplitGrouper.java:getGroupedSplits(539)) - Number of splits desired: 1 created: 1 splitsProcessed: 2
2016-05-18 00:57:55,215 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(308)) - Succeeded InputInitializer for Input: ED4172E4561A4AE5933B8AFD35FA9D0F on vertex vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,215 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:55,216 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1644)) - Got updated RootInputsSpecs: {ED4172E4561A4AE5933B8AFD35FA9D0F=forAllWorkUnits=true, update=[1]}
2016-05-18 00:57:55,216 INFO impl.VertexImpl (VertexImpl.java:setParallelismWrapper(1664)) - Vertex vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] parallelism set to 1
2016-05-18 00:57:55,216 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(2972)) - Cannot init vertex: vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 1 initWaitsForRootInitializers: true
2016-05-18 00:57:55,216 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A90BB0BD5F2640B6B3D0FC04C551A31C, vertexId=vertex_1463533075047_0001_1_00, initRequestedTime=1463533075206, initedTime=1463533075216, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1, initGeneratedEventsCount=1
2016-05-18 00:57:55,216 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] with distanceFromRoot: 0
2016-05-18 00:57:55,217 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] transitioned from INITIALIZING to INITED due to event V_INPUT_DATA_INFORMATION
2016-05-18 00:57:55,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533075047_0001_1_00, startRequestedTime=1463533075207, startedTime=1463533075217
2016-05-18 00:57:55,217 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:VERTEX_CONFIGURE_DONE]: vertexId=vertex_1463533075047_0001_1_00, reconfigureDoneTime=0, numTasks=1, vertexLocationHint=org.apache.tez.dag.api.VertexLocationHint@b8338fe7, edgeManagersCount=0, rootInputSpecUpdateCount=1, setParallelismCalledFlag=true
2016-05-18 00:57:55,217 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:55,217 INFO impl.VertexImpl (VertexImpl.java:transition(2932)) - Source vertex started: vertex_1463533075047_0001_1_00 for vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] numStartedSources: 1 numSources: 1
2016-05-18 00:57:55,217 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(2814)) - Triggering start event for vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] with distanceFromRoot: 1
2016-05-18 00:57:55,218 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463533075047_0001_1_01, startRequestedTime=1463533075217, startedTime=1463533075217
2016-05-18 00:57:55,218 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] transitioned from INITED to RUNNING due to event V_START
2016-05-18 00:57:55,218 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(100)) - Starting 1 in A90BB0BD5F2640B6B3D0FC04C551A31C
2016-05-18 00:57:55,219 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,219 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:TASK_STARTED]: vertexName=A90BB0BD5F2640B6B3D0FC04C551A31C, taskId=task_1463533075047_0001_1_00_000000, scheduledTime=1463533075219, launchTime=1463533075219
2016-05-18 00:57:55,219 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(519)) - OnVertexStarted vertex: A3D788584351473E8D00CD9B27867E14 with 0 source tasks and 1 pending tasks
2016-05-18 00:57:55,219 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,220 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(1051)) - Received configured notification : CONFIGURED for vertex: A90BB0BD5F2640B6B3D0FC04C551A31C in vertex: A3D788584351473E8D00CD9B27867E14 numBipartiteSourceTasks: 1
2016-05-18 00:57:55,220 INFO node.PerSourceNodeTracker (PerSourceNodeTracker.java:nodeSeen(74)) - Adding new node 127.0.0.1:0 to nodeTracker 0
2016-05-18 00:57:55,221 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533075047_0001_00_000001, launchTime=1463533075221
2016-05-18 00:57:55,222 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533075047_0001_00_000001
2016-05-18 00:57:55,222 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533075047_0001_1_00_000000_0] started. Is using containerId: [container_1463533075047_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-18 00:57:55,223 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A90BB0BD5F2640B6B3D0FC04C551A31C, taskAttemptId=attempt_1463533075047_0001_1_00_000000_0, startTime=1463533075222, containerId=container_1463533075047_0001_00_000001, nodeId=127.0.0.1:0
2016-05-18 00:57:55,223 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533075047_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533075047_0001_1_00_000000_0
2016-05-18 00:57:55 Starting to run new task attempt: attempt_1463533075047_0001_1_00_000000_0
2016-05-18 00:57:55,224 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533075047_0001_1_00_000000_0
2016-05-18 00:57:55,224 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:55,224 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:55,225 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6] every/(1/1) ...blyplatform/ALL_ARGS_SWAP, VertexName: A90BB0BD5F2640B6B3D0FC04C551A31C, VertexParallelism: 1, TaskAttemptID:attempt_1463533075047_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=ED4172E4561A4AE5933B8AFD35FA9D0F, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A3D788584351473E8D00CD9B27867E14, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-18 00:57:55,225 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:55,225 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533075047_0001_1_00_000000_0
2016-05-18 00:57:55,226 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(496)) - TaskAttempt:attempt_1463533075047_0001_1_00_000000_0 sent events: (0-1).
2016-05-18 00:57:55,226 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533075047_0001_1_00_000000_0, eventCount=1 fromEventId=0 nextFromEventId=0
2016-05-18 00:57:55,227 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] submitted tez dag to app master: application_1463533075047_0001, with dag id: dag_1463533075047_0001_1
2016-05-18 00:57:55,230 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:55,232 INFO input.MRInput (MRInput.java:initialize(440)) - ED4172E4561A4AE5933B8AFD35FA9D0F using newmapreduce API=false, split via event=true, numPhysicalInputs=1
2016-05-18 00:57:55,233 INFO input.MRInput (MRInput.java:initializeInternal(494)) - Initialized MRInput: ED4172E4561A4AE5933B8AFD35FA9D0F
2016-05-18 00:57:55,234 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A90BB0BD5F2640B6B3D0FC04C551A31C, ordinal: 0
2016-05-18 00:57:55,234 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A90BB0BD5F2640B6B3D0FC04C551A31C, mem on start (mb), free: 69, total: 369, max: 672
2016-05-18 00:57:55,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:55,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:55,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:55,234 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:55,235 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[A3D788584351473E8D00CD9B27867E14:OUTPUT:104857600:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput], [ED4172E4561A4AE5933B8AFD35FA9D0F:INPUT:0:org.apache.tez.mapreduce.input.MRInput]
2016-05-18 00:57:55,235 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:55,235 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:55,235 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-18 00:57:55,235 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[A3D788584351473E8D00CD9B27867E14:org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput:OUTPUT:104857600:104857600], [ED4172E4561A4AE5933B8AFD35FA9D0F:org.apache.tez.mapreduce.input.MRInput:INPUT:0:0]
2016-05-18 00:57:55,236 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:55,236 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: ED4172E4561A4AE5933B8AFD35FA9D0F
2016-05-18 00:57:55,236 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: ED4172E4561A4AE5933B8AFD35FA9D0F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:55,236 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:55,236 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:55,237 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:55,237 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533075047_0001_1_00_000000_0
2016-05-18 00:57:55,238 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(174)) - Processing split: TezGroupedSplit{wrappedSplits=[file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:0+25, file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt:25+26], wrappedInputFormatName='org.apache.hadoop.mapred.TextInputFormat', locations=[localhost], rack='null', length=51}
2016-05-18 00:57:55,238 INFO input.MRInput (MRInput.java:initFromEventInternal(653)) - ED4172E4561A4AE5933B8AFD35FA9D0F initialized RecordReader from event
2016-05-18 00:57:55,239 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:55,240 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"] streamed: true, id: ED4172E4561A4AE5933B8AFD35FA9D0F
2016-05-18 00:57:55,240 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(every)[by:[{?}:ALL]], id: 01BB8CFFD07F4A5A8EC590E91D1A9C2A
2016-05-18 00:57:55,240 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(every)[by:[{?}:ALL]] 01BB8CFFD07F4A5A8EC590E91D1A9C2A
2016-05-18 00:57:55,241 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:55,243 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:55,243 INFO impl.ExternalSorter (ExternalSorter.java:<init>(199)) - A3D788584351473E8D00CD9B27867E14 using: memoryMb=100, keySerializerClass=class cascading.tuple.io.KeyTuple, valueSerializerClass=cascading.tuple.hadoop.io.ValueTupleSerializer@7c44b314, comparator=cascading.tuple.hadoop.util.TupleComparator@24925c9, partitioner=cascading.tuple.tez.util.TuplePartitioner, serialization=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-18 00:57:55,243 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(170)) - Setting up PipelinedSorter for A3D788584351473E8D00CD9B27867E14: , UsingHashComparator=false
2016-05-18 00:57:55,264 INFO impl.PipelinedSorter (PipelinedSorter.java:allocateSpace(245)) - Newly allocated block size=104857600, index=0, Number of buffers=1, currentAllocatableMemory=0, currentBufferSize=104857600, total=104857600
2016-05-18 00:57:55,264 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(189)) - Pre allocating rest of memory buffers upfront
2016-05-18 00:57:55,264 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(207)) - Setting up PipelinedSorter for A3D788584351473E8D00CD9B27867E14: , UsingHashComparator=false#blocks=1, maxMemUsage=104857600, lazyAllocateMem=false, minBlockSize=2097152000, initial BLOCK_SIZE=104857600, finalMergeEnabled=true, pipelinedShuffle=false, sendEmptyPartitions=true, tez.runtime.io.sort.mb=100
2016-05-18 00:57:55,264 INFO impl.PipelinedSorter (PipelinedSorter.java:<init>(882)) - A3D788584351473E8D00CD9B27867E14: reserved.remaining()=104857600, reserved.metasize=16777216
2016-05-18 00:57:55,265 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/classes/test/data/nums.20.txt"]
2016-05-18 00:57:55,265 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A90BB0BD5F2640B6B3D0FC04C551A31C, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:55,268 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(79)) - Closing task, taskAttemptId=attempt_1463533075047_0001_1_00_000000_0
2016-05-18 00:57:55,268 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A90BB0BD5F2640B6B3D0FC04C551A31C, mem on close (mb), free: 161, total: 369, max: 672
2016-05-18 00:57:55,268 INFO impl.PipelinedSorter (PipelinedSorter.java:flush(629)) - A3D788584351473E8D00CD9B27867E14: Starting flush of map output
2016-05-18 00:57:55,268 INFO impl.PipelinedSorter (PipelinedSorter.java:end(995)) - A3D788584351473E8D00CD9B27867E14: Span0.length = 20, perItem = 6
2016-05-18 00:57:55,269 INFO impl.PipelinedSorter (PipelinedSorter.java:sort(904)) - A3D788584351473E8D00CD9B27867E14: done sorting span=0, length=20, time=1
2016-05-18 00:57:55,269 INFO impl.PipelinedSorter (PipelinedSorter.java:ready(1297)) - A3D788584351473E8D00CD9B27867E14: Heap = SpanIterator<0:19> (span=Span[16777216,131]),
2016-05-18 00:57:55,270 INFO impl.PipelinedSorter (PipelinedSorter.java:spill(546)) - A3D788584351473E8D00CD9B27867E14: Spilling to /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001_wd/localmode-local-dir/output/attempt_1463533075047_0001_1_00_000000_0_10338_0/file.out
2016-05-18 00:57:55,274 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:close(387)) - Joining on EventRouter
2016-05-18 00:57:55 Completed running task attempt: attempt_1463533075047_0001_1_00_000000_0
2016-05-18 00:57:55,274 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(86)) - Task completed, taskAttemptId=attempt_1463533075047_0001_1_00_000000_0, askedToStop=false
2016-05-18 00:57:55,274 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533075047_0001_1_00_000000_0, stopRequested=false
2016-05-18 00:57:55,275 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533075047_0001_1_00_000000_0: Counters: 18 [[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1][cascading.flow.SliceCounters Process_Begin_Time=1463533075265, Process_Duration=3, Process_End_Time=1463533075268, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=20]]
2016-05-18 00:57:55,275 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533075047_0001_1_00_000000_0 : TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:55,275 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=SUCCESS, error=null, containerShutdownRequested=false}
2016-05-18 00:57:55,275 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A90BB0BD5F2640B6B3D0FC04C551A31C, taskAttemptId=attempt_1463533075047_0001_1_00_000000_0, creationTime=1463533075219, allocationTime=1463533075221, startTime=1463533075222, finishTime=1463533075275, timeTaken=53, status=SUCCEEDED, errorEnum=, diagnostics=
2016-05-18 00:57:55,275 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend:
2016-05-18 00:57:55,275 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:55,276 INFO impl.TaskImpl (TaskImpl.java:transition(1057)) - Task succeeded with attempt attempt_1463533075047_0001_1_00_000000_0
2016-05-18 00:57:55,276 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:TASK_FINISHED]: vertexName=A90BB0BD5F2640B6B3D0FC04C551A31C, taskId=task_1463533075047_0001_1_00_000000, startTime=1463533075222, finishTime=1463533075276, timeTaken=54, status=SUCCEEDED, successfulAttemptID=attempt_1463533075047_0001_1_00_000000_0, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533075265, Process_Duration=3, Process_End_Time=1463533075268, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:55,276 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C], tasks=1, failed=0, killed=0, success=1, completed=1, commits=0, err=null
2016-05-18 00:57:55,276 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2037)) - All tasks have succeeded, vertex:vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C]
2016-05-18 00:57:55,276 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533075047_0001_00_000001
2016-05-18 00:57:55,276 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:VERTEX_FINISHED]: vertexName=A90BB0BD5F2640B6B3D0FC04C551A31C, vertexId=vertex_1463533075047_0001_1_00, initRequestedTime=1463533075206, initedTime=1463533075216, startRequestedTime=1463533075207, startedTime=1463533075217, finishTime=1463533075276, timeTaken=59, status=SUCCEEDED, diagnostics=, counters=Counters: 19, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533075265, Process_Duration=3, Process_End_Time=1463533075268, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20, vertexStats=firstTaskStartTime=1463533075222, firstTasksToStart=[ task_1463533075047_0001_1_00_000000 ], lastTaskFinishTime=1463533075275, lastTasksToFinish=[ task_1463533075047_0001_1_00_000000 ], minTaskDuration=53, maxTaskDuration=53, avgTaskDuration=53.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463533075047_0001_1_00_000000 ], longestDurationTasks=[ task_1463533075047_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-18 00:57:55,277 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-18 00:57:55,277 INFO impl.VertexImpl (VertexImpl.java:transition(3279)) - Source task attempt completed for vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] attempt: attempt_1463533075047_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-18 00:57:55,277 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533075047_0001_1_00 [A90BB0BD5F2640B6B3D0FC04C551A31C] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-18 00:57:55,277 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(920)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A3D788584351473E8D00CD9B27867E14
2016-05-18 00:57:55,277 INFO impl.Edge (Edge.java:routingToBegin(232)) - Routing to begin for edge: EdgeInfo: sourceVertexName=A90BB0BD5F2640B6B3D0FC04C551A31C, destinationVertexName=A3D788584351473E8D00CD9B27867E14. EdgeProperty: { SCATTER_GATHER : org.apache.tez.runtime.library.input.OrderedGroupedKVInput >> PERSISTED >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput >> NullEdgeManager } onDemandRouting: true
2016-05-18 00:57:55,277 INFO impl.VertexImpl (VertexImpl.java:unsetTasksNotYetScheduled(1466)) - Routing pending task events for vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,277 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=null
2016-05-18 00:57:55,278 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:TASK_STARTED]: vertexName=A3D788584351473E8D00CD9B27867E14, taskId=task_1463533075047_0001_1_01_000000, scheduledTime=1463533075278, launchTime=1463533075278
2016-05-18 00:57:55,278 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTaskEx(106)) - Scheduled vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,279 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533075047_0001_00_000001, stoppedTime=1463533075279, exitStatus=0
2016-05-18 00:57:55,279 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(262)) - Ignoring stop request for containerId container_1463533075047_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-18 00:57:55,280 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:N/A][Event:CONTAINER_LAUNCHED]: containerId=container_1463533075047_0001_00_000002, launchTime=1463533075280
2016-05-18 00:57:55,280 INFO task.ContainerReporter (ContainerReporter.java:callInternal(57)) - Attempting to fetch new task for container container_1463533075047_0001_00_000002
2016-05-18 00:57:55,281 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533075047_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463533075047_0001_1_01_000000_0
2016-05-18 00:57:55,281 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1322)) - TaskAttempt: [attempt_1463533075047_0001_1_01_000000_0] started. Is using containerId: [container_1463533075047_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-18 00:57:55,281 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A3D788584351473E8D00CD9B27867E14, taskAttemptId=attempt_1463533075047_0001_1_01_000000_0, startTime=1463533075281, containerId=container_1463533075047_0001_00_000002, nodeId=127.0.0.1:0
2016-05-18 00:57:55 Starting to run new task attempt: attempt_1463533075047_0001_1_01_000000_0
2016-05-18 00:57:55,281 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: attempt_1463533075047_0001_1_01_000000_0
2016-05-18 00:57:55,282 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:55,282 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(306)) - Refreshing UGI since Credentials have changed. Credentials : #Tokens=1, #SecretKeys=0
2016-05-18 00:57:55,282 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(166)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6] every/(1/1) ...blyplatform/ALL_ARGS_SWAP, VertexName: A3D788584351473E8D00CD9B27867E14, VertexParallelism: 1, TaskAttemptID:attempt_1463533075047_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A90BB0BD5F2640B6B3D0FC04C551A31C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=743B00DF4205494EB060C46586059969, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-18 00:57:55,282 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(93)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072, allocatorClassName=org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-18 00:57:55,283 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(65)) - Initializing task, taskAttemptId=attempt_1463533075047_0001_1_01_000000_0
2016-05-18 00:57:55,285 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(339)) - Initial Memory required for SHUFFLE_BUFFER=211392912 based on INPUT_BUFFER_FACTOR=0.9, for final merged output=0, using factor: 0.0
2016-05-18 00:57:55,285 INFO impl.VertexImpl (VertexImpl.java:getTaskAttemptTezEvents(3662)) - Sending attempt_1463533075047_0001_1_01_000000_0 1 events [0,1) total 1 vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,285 INFO task.TaskReporter (TaskReporter.java:heartbeat(292)) - Routing events from heartbeat response to task, currentTaskAttemptId=attempt_1463533075047_0001_1_01_000000_0, eventCount=1 fromEventId=0 nextFromEventId=1
2016-05-18 00:57:55,286 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-18 00:57:55,287 INFO app.TezTaskCommunicatorImpl (TezTaskCommunicatorImpl.java:getContainerTask(416)) - Container with id: container_1463533075047_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-18 00:57:55,287 INFO task.ContainerReporter (ContainerReporter.java:callInternal(67)) - Got TaskUpdate for containerId= container_1463533075047_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-18 00:57:55,287 INFO task.TezChild (TezChild.java:run(235)) - ContainerTask returned shouldDie=true for container container_1463533075047_0001_00_000001, Exiting
2016-05-18 00:57:55,287 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533075047_0001_00_000001
2016-05-18 00:57:55,287 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533075047_0001_00_000001
2016-05-18 00:57:55,289 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533075047_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-18 00:57:55,290 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(289)) - Container: container_1463533075047_0001_00_000001 completed successfully
2016-05-18 00:57:55,290 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A3D788584351473E8D00CD9B27867E14, ordinal: 1
2016-05-18 00:57:55,290 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A3D788584351473E8D00CD9B27867E14, mem on start (mb), free: 156, total: 369, max: 672
2016-05-18 00:57:55,290 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(562)) - Initialized processor
2016-05-18 00:57:55,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 2 initializers to finish
2016-05-18 00:57:55,291 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(239)) - Waiting for 1 initializers to finish
2016-05-18 00:57:55,293 INFO output.MROutput (MROutput.java:initialize(421)) - 743B00DF4205494EB060C46586059969: outputFormat=org.apache.hadoop.mapred.TextOutputFormat, using newmapreduce API=false
2016-05-18 00:57:55,293 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(252)) - All initializers finished
2016-05-18 00:57:55,293 INFO resources.MemoryDistributor (MemoryDistributor.java:logInitialRequests(275)) - InitialRequests=[743B00DF4205494EB060C46586059969:OUTPUT:0:org.apache.tez.mapreduce.output.MROutput], [A90BB0BD5F2640B6B3D0FC04C551A31C:INPUT:211392912:org.apache.tez.runtime.library.input.OrderedGroupedKVInput]
2016-05-18 00:57:55,293 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:populateTypeScaleMap(239)) - ScaleRatiosUsed=[PARTITIONED_UNSORTED_OUTPUT:1][UNSORTED_OUTPUT:1][UNSORTED_INPUT:1][SORTED_OUTPUT:12][SORTED_MERGED_INPUT:12][PROCESSOR:1][OTHER:1]
2016-05-18 00:57:55,294 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(266)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-18 00:57:55,294 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 211392912, TotalRequestedScaled: 1.9513191876923078E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.30
2016-05-18 00:57:55,294 INFO resources.MemoryDistributor (MemoryDistributor.java:logFinalAllocations(297)) - Allocations=[743B00DF4205494EB060C46586059969:org.apache.tez.mapreduce.output.MROutput:OUTPUT:0:0], [A90BB0BD5F2640B6B3D0FC04C551A31C:org.apache.tez.runtime.library.input.OrderedGroupedKVInput:INPUT:211392912:211392912]
2016-05-18 00:57:55,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(274)) - Starting Inputs/Outputs
2016-05-18 00:57:55,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(287)) - Input: A90BB0BD5F2640B6B3D0FC04C551A31C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-18 00:57:55,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(308)) - Num IOs determined for AutoStart: 1
2016-05-18 00:57:55,294 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(147)) - A90BB0BD5F2640B6B3D0FC04C551A31C: Shuffle assigned with 1 inputs, codec: None, ifileReadAhead: true
2016-05-18 00:57:55,295 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(273)) - A90BB0BD5F2640B6B3D0FC04C551A31C: MergerManager: memoryLimit=211392912, maxSingleShuffleLimit=52848228, mergeThreshold=190253616, ioSortFactor=100, postMergeMem=0, memToMemMergeOutputsThreshold=100
2016-05-18 00:57:55,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(310)) - Waiting for 1 IOs to start
2016-05-18 00:57:55,295 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(353)) - ShuffleScheduler running for sourceVertex: A90BB0BD5F2640B6B3D0FC04C551A31C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=15, maxTaskOutputAtOnce=20, numFetchers=1, hostFailureFraction=0.2, minFailurePerHost=4, maxAllowedFailedFetchFraction=0.5, maxStallTimeFraction=0.5, minReqProgressFraction=0.5, checkFailedFetchSinceLastCompletion=true
2016-05-18 00:57:55,296 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:_callInternal(474)) - Started Input with src edge: A90BB0BD5F2640B6B3D0FC04C551A31C
2016-05-18 00:57:55,296 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(323)) - AutoStartComplete
2016-05-18 00:57:55,296 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:run(70)) - Running task, taskAttemptId=attempt_1463533075047_0001_1_01_000000_0
2016-05-18 00:57:55,297 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A90BB0BD5F2640B6B3D0FC04C551A31C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0
2016-05-18 00:57:55,298 INFO orderedgrouped.ShuffleScheduler (ShuffleUtils.java:logIndividualFetchComplete(489)) - Completed fetch for attempt: {0, 0, attempt_1463533075047_0001_1_00_000000_0_10338} to DISK_DIRECT, csize=181, dsize=177, EndTime=1463533075298, TimeTaken=1, Rate=0.17 MB/s
2016-05-18 00:57:55,298 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(551)) - All inputs fetched for input vertex : A90BB0BD5F2640B6B3D0FC04C551A31C
2016-05-18 00:57:55,298 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:55,298 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:callInternal(1289)) - Shutting down FetchScheduler for input: A90BB0BD5F2640B6B3D0FC04C551A31C, wasInterrupted=false
2016-05-18 00:57:55,299 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(618)) - copy(1 (spillsFetched=1) of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-18 00:57:55,299 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1005)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-18 00:57:55,299 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:onSuccess(1328)) - A90BB0BD5F2640B6B3D0FC04C551A31C: Already shutdown. Ignoring fetch complete
2016-05-18 00:57:55,300 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:55,300 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1103)) - Disk file=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001_wd/localmode-local-dir/output/attempt_1463533075047_0001_1_00_000000_0_10338/file.out, len=181, isLocal=true
2016-05-18 00:57:55,300 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1115)) - Merging 1 files, 181 bytes from disk
2016-05-18 00:57:55,300 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(1130)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-18 00:57:55,300 INFO impl.TezMerger (TezMerger.java:merge(644)) - Merging 1 sorted segments
2016-05-18 00:57:55,300 INFO impl.TezMerger (TezMerger.java:merge(749)) - Down to the last merge-pass, with 1 segments left of total size: 165 bytes
2016-05-18 00:57:55,301 INFO orderedgrouped.Shuffle (Shuffle.java:callInternal(332)) - merge complete for input vertex : A90BB0BD5F2640B6B3D0FC04C551A31C
2016-05-18 00:57:55,301 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(417)) - A90BB0BD5F2640B6B3D0FC04C551A31C: Shuffle Runner thread complete
2016-05-18 00:57:55,301 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(every)[by:[{?}:ALL]]
2016-05-18 00:57:55,301 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(every)[by:[{?}:ALL]] streamed: true, id: 01BB8CFFD07F4A5A8EC590E91D1A9C2A
2016-05-18 00:57:55,302 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_SWAP"], id: 743B00DF4205494EB060C46586059969
2016-05-18 00:57:55,302 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_SWAP"]
2016-05-18 00:57:55,302 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(every)[by:[{?}:ALL]] 01BB8CFFD07F4A5A8EC590E91D1A9C2A, for 1 inputs
2016-05-18 00:57:55,302 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A3D788584351473E8D00CD9B27867E14, all 1 inputs ready in: 00:00:00.000
2016-05-18 00:57:55,303 INFO util.BaseTupleComparator (BaseTupleComparator.java:setConf(57)) - enabling raw byte comparison and ordering
2016-05-18 00:57:55,303 INFO input.OrderedGroupedKVInput (OrderedGroupedKVInput.java:createValuesIterator(292)) - A90BB0BD5F2640B6B3D0FC04C551A31C: creating ValuesIterator with comparator=cascading.tuple.hadoop.util.TupleComparator, keyClass=cascading.tuple.io.KeyTuple, valClass=cascading.tuple.io.ValueTuple
2016-05-18 00:57:55,305 ERROR element.TrapHandler (TrapHandler.java:handleReThrowableException(115)) - caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:55,307 ERROR element.TezGroupByGate (TezGroupByGate.java:reduce(91)) - caught throwable
cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
2016-05-18 00:57:55,308 INFO output.MROutput (MROutput.java:close(526)) - 743B00DF4205494EB060C46586059969 closed
2016-05-18 00:57:55,309 INFO impl.TaskImpl (TaskImpl.java:canCommit(671)) - attempt_1463533075047_0001_1_01_000000_0 given a go for committing the task output.
2016-05-18 00:57:55,309 INFO task.TaskRunner2Callable (TaskRunner2Callable.java:callInternal(104)) - Cleaning up task attempt_1463533075047_0001_1_01_000000_0, stopRequested=false
2016-05-18 00:57:55,309 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(813)) - Final Counters for attempt_1463533075047_0001_1_01_000000_0: Counters: 33 [[org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=0, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=4, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=3, MERGE_PHASE_TIME=6, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463533075302, Process_Duration=7, Process_End_Time=1463533075309, Read_Duration=0, Tuples_Read=1]]
2016-05-18 00:57:55,309 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(817)) - Joining on EventRouter
2016-05-18 00:57:55,310 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A3D788584351473E8D00CD9B27867E14, mem on close (mb), free: 152, total: 369, max: 672
2016-05-18 00:57:55,310 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(851)) - Closed processor for vertex=A3D788584351473E8D00CD9B27867E14, index=1, interruptedStatus=false
2016-05-18 00:57:55,310 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(278)) - Shutting down Shuffle for source: A90BB0BD5F2640B6B3D0FC04C551A31C
2016-05-18 00:57:55,310 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:logProgress(76)) - A90BB0BD5F2640B6B3D0FC04C551A31C: numDmeEventsSeen=1, numDmeEventsSeenWithNoData=0, numObsoletionEventsSeen=0, updateOnClose
2016-05-18 00:57:55,310 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(887)) - Closed input for vertex=A3D788584351473E8D00CD9B27867E14, sourceVertex=A90BB0BD5F2640B6B3D0FC04C551A31C, interruptedStatus=true
2016-05-18 00:57:55,310 INFO output.MROutput (MROutput.java:close(526)) - 743B00DF4205494EB060C46586059969 closed
2016-05-18 00:57:55,311 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(911)) - Closed input for vertex=A3D788584351473E8D00CD9B27867E14, sourceVertex=743B00DF4205494EB060C46586059969, interruptedStatus=true
2016-05-18 00:57:55,311 INFO task.TezTaskRunner2 (TezTaskRunner2.java:logAndReturnEndResult(473)) - TaskRunnerResult for attempt_1463533075047_0001_1_01_000000_0 : TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], containerShutdownRequested=false}
2016-05-18 00:57:55,311 INFO task.TezChild (TezChild.java:run(265)) - TaskRunner2Result: TaskRunner2Result{endReason=TASK_ERROR, error=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], containerShutdownRequested=false}
2016-05-18 00:57:55,311 INFO task.TezChild (TezChild.java:shutdown(381)) - Shutdown invoked for container container_1463533075047_0001_00_000002
2016-05-18 00:57:55,311 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A3D788584351473E8D00CD9B27867E14, taskAttemptId=attempt_1463533075047_0001_1_01_000000_0, creationTime=1463533075278, allocationTime=1463533075279, startTime=1463533075281, finishTime=1463533075311, timeTaken=30, status=FAILED, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Failure while running task: attempt_1463533075047_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
, counters=Counters: 33, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=2, REDUCE_INPUT_RECORDS=1, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=2, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, OUTPUT_RECORDS=0, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=181, SHUFFLE_BYTES=181, SHUFFLE_BYTES_DECOMPRESSED=177, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=181, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=3, MERGE_PHASE_TIME=6, FIRST_EVENT_RECEIVED=2, LAST_EVENT_RECEIVED=2, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463533075302, Process_Duration=7, Process_End_Time=1463533075309, Read_Duration=0, Tuples_Read=1
2016-05-18 00:57:55,312 INFO impl.TaskImpl (TaskImpl.java:transition(1170)) - Failing task: task_1463533075047_0001_1_01_000000, currentFailedAttempts: 1, maxFailedAttempts: 1
2016-05-18 00:57:55,312 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:TASK_FINISHED]: vertexName=A3D788584351473E8D00CD9B27867E14, taskId=task_1463533075047_0001_1_01_000000, startTime=1463533075281, finishTime=1463533075312, timeTaken=31, status=FAILED, successfulAttemptID=null, diagnostics=TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533075047_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
], counters=Counters: 0
2016-05-18 00:57:55,312 INFO impl.VertexImpl (VertexImpl.java:transition(3353)) - Failing vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] because task failed: task_1463533075047_0001_1_01_000000
2016-05-18 00:57:55,312 INFO impl.VertexImpl (VertexImpl.java:tryEnactKill(2102)) - Killing tasks in vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] due to trigger: OWN_TASK_FAILURE
2016-05-18 00:57:55,311 INFO task.TezChild (TezChild.java:shutdown(383)) - Shutting down container container_1463533075047_0001_00_000002
2016-05-18 00:57:55,312 INFO impl.VertexImpl (VertexImpl.java:checkTasksForCompletion(2024)) - Task Completion: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14], tasks=1, failed=1, killed=0, success=0, completed=1, commits=0, err=OWN_TASK_FAILURE
2016-05-18 00:57:55,313 INFO impl.VertexImpl (VertexImpl.java:finishWithTerminationCause(2084)) - Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
2016-05-18 00:57:55,313 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3087)) - Invoking committer abort for vertex, vertexId=vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,313 INFO impl.VertexImpl (VertexImpl.java:run(3095)) - Invoking committer abort for output=743B00DF4205494EB060C46586059969, vertexId=vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,313 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(285)) - ContainerExecutionResult for: container_1463533075047_0001_00_000002 = ContainerExecutionResult{exitStatus=EXECUTION_FAILURE, throwable=cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS], errorMessage='TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]'}
2016-05-18 00:57:55,313 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(294)) - Container: container_1463533075047_0001_00_000002 completed but with errors
2016-05-18 00:57:55,314 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:VERTEX_FINISHED]: vertexName=A3D788584351473E8D00CD9B27867E14, vertexId=vertex_1463533075047_0001_1_01, initRequestedTime=1463533075207, initedTime=1463533075210, startRequestedTime=1463533075217, startedTime=1463533075217, finishTime=1463533075314, timeTaken=97, status=FAILED, diagnostics=Task failed, taskId=task_1463533075047_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533075047_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]]
Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0
Vertex vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] killed/failed due to:OWN_TASK_FAILURE, counters=Counters: 0, vertexStats=firstTaskStartTime=1463533075281, firstTasksToStart=[ task_1463533075047_0001_1_01_000000 ], lastTaskFinishTime=1463533075311, lastTasksToFinish=[ task_1463533075047_0001_1_01_000000 ], minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=1, numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=1, numCompletedTasks=1}
2016-05-18 00:57:55,314 INFO impl.VertexImpl (VertexImpl.java:handle(1823)) - vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] transitioned from RUNNING to FAILED due to event V_TASK_COMPLETED
2016-05-18 00:57:55,314 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463533075047_0001_00_000002, stoppedTime=1463533075314, exitStatus=-1000
2016-05-18 00:57:55,315 INFO node.AMNodeImpl (AMNodeImpl.java:transition(298)) - Attempt failed on node: 127.0.0.1:0 TA: attempt_1463533075047_0001_1_01_000000_0 failed: true container: container_1463533075047_0001_00_000002 numFailedTAs: 0
2016-05-18 00:57:55,315 INFO container.AMContainerImpl (AMContainerImpl.java:transition(993)) - Container container_1463533075047_0001_00_000002 exited with diagnostics set to TaskExecutionFailure: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
2016-05-18 00:57:55,315 INFO impl.DAGImpl (DAGImpl.java:transition(1927)) - Vertex vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] completed., numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2
2016-05-18 00:57:55,315 INFO impl.DAGImpl (DAGImpl.java:checkVerticesForCompletion(1261)) - Checking vertices for DAG completion, numCompletedVertices=2, numSuccessfulVertices=1, numFailedVertices=1, numKilledVertices=0, numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
2016-05-18 00:57:55,315 INFO impl.DAGImpl (DAGImpl.java:finishWithTerminationCause(1350)) - DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
2016-05-18 00:57:55,316 INFO recovery.RecoveryService (RecoveryService.java:handle(327)) - DAG completed, dagId=dag_1463533075047_0001_1, queueSize=0
2016-05-18 00:57:55,317 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(115)) - [HISTORY][DAG:dag_1463533075047_0001_1][Event:DAG_FINISHED]: dagId=dag_1463533075047_0001_1, startTime=1463533075205, finishTime=1463533075316, timeTaken=111, status=FAILED, diagnostics=Vertex failed, vertexName=A3D788584351473E8D00CD9B27867E14, vertexId=vertex_1463533075047_0001_1_01, diagnostics=[Task failed, taskId=task_1463533075047_0001_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task: attempt_1463533075047_0001_1_01_000000_0:cascading.pipe.OperatorException: [every][cascading.detail.EveryAssemblyFactory.createAssembly(EveryAssemblyFactory.java:36)] operator Every failed completing operation: TestAggregator[decl:ARGS]
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:161)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:41)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:51)
at cascading.flow.stream.duct.OpenReducingDuct.receive(OpenReducingDuct.java:28)
at cascading.flow.hadoop.stream.HadoopGroupGate.accept(HadoopGroupGate.java:147)
at cascading.flow.tez.stream.element.TezGroupByGate.reduce(TezGroupByGate.java:83)
at cascading.flow.tez.stream.element.TezGroupGate.run(TezGroupGate.java:114)
at cascading.flow.tez.FlowProcessor.run(FlowProcessor.java:165)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:351)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:71)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:59)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:59)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: current entry and given tuple must be same length
at cascading.tuple.TupleEntry.setCanonicalTuple(TupleEntry.java:405)
at cascading.TestAggregator.complete(TestAggregator.java:112)
at cascading.flow.stream.element.AggregatorEveryStage.completeGroup(AggregatorEveryStage.java:153)
... 20 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0, counters=Counters: 24, org.apache.tez.common.counters.DAGCounter, NUM_FAILED_TASKS=1, NUM_SUCCEEDED_TASKS=1, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, AM_CPU_MILLISECONDS=190, AM_GC_TIME_MILLIS=8, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=20, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=20, OUTPUT_BYTES=131, OUTPUT_BYTES_WITH_OVERHEAD=177, OUTPUT_BYTES_PHYSICAL=181, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_CHUNK_COUNT=1, cascading.flow.SliceCounters, Process_Begin_Time=1463533075265, Process_Duration=3, Process_End_Time=1463533075268, Read_Duration=2, Tuples_Read=20, Tuples_Written=20, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=20
2016-05-18 00:57:55,317 INFO impl.VertexImpl (VertexImpl.java:abortVertex(3082)) - Ignoring multiple aborts for vertex: vertex_1463533075047_0001_1_01 [A3D788584351473E8D00CD9B27867E14]
2016-05-18 00:57:55,317 INFO impl.DAGImpl (DAGImpl.java:finished(1405)) - DAG: dag_1463533075047_0001_1 finished with state: FAILED
2016-05-18 00:57:55,317 INFO impl.DAGImpl (DAGImpl.java:handle(1170)) - dag_1463533075047_0001_1 transitioned from RUNNING to FAILED due to event DAG_VERTEX_COMPLETED
2016-05-18 00:57:55,317 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(260)) - Ignoring stop request for containerId: container_1463533075047_0001_00_000002
2016-05-18 00:57:55 Completed Dag: dag_1463533075047_0001_1
2016-05-18 00:57:55,318 INFO app.DAGAppMaster (DAGAppMaster.java:handle(751)) - DAG completed, dagId=dag_1463533075047_0001_1, dagState=FAILED
2016-05-18 00:57:55,318 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(188)) - Redirecting log file based on addend: dag_1463533075047_0001_1_post
2016-05-18 00:57:55,318 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(203)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-18 00:57:55,318 INFO app.DAGAppMaster (DAGAppMaster.java:handle(800)) - Central Dispatcher queue size after DAG completion, before cleanup: 1
2016-05-18 00:57:55,319 INFO app.DAGAppMaster (DAGAppMaster.java:handle(802)) - Waiting for next DAG to be submitted.
2016-05-18 00:57:55,319 INFO app.DAGAppMaster (DAGAppMaster.java:handle(842)) - Cleaning up DAG: name=[E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6] every/(1/1) ...blyplatform/ALL_ARGS_SWAP, with id=dag_1463533075047_0001_1
2016-05-18 00:57:55,319 INFO app.DAGAppMaster (DAGAppMaster.java:handle(852)) - Completed cleanup for DAG: name=[E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6] every/(1/1) ...blyplatform/ALL_ARGS_SWAP, with id=dag_1463533075047_0001_1
2016-05-18 00:57:55,321 INFO client.TezClient (TezClient.java:stop(566)) - Shutting down Tez Session, sessionName=(1/1) ...blyplatform/ALL_ARGS_SWAP, applicationId=application_1463533075047_0001
2016-05-18 00:57:55,323 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(124)) - Received message to shutdown AM
2016-05-18 00:57:55,323 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:setShouldUnregisterFlag(895)) - TaskScheduler notified that it should unregister from RM
2016-05-18 00:57:55,323 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1301)) - No current running DAG, shutting down the AM
2016-05-18 00:57:55,323 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(907)) - DAGAppMasterShutdownHandler invoked
2016-05-18 00:57:55,323 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(916)) - Handling DAGAppMaster shutdown
2016-05-18 00:57:55,323 INFO app.DAGAppMaster (DAGAppMaster.java:run(936)) - Sleeping for 5 seconds before shutting down
2016-05-18 00:57:55,323 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(84)) - Stopping HistoryEventHandler
2016-05-18 00:57:55,324 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(196)) - Stopping RecoveryService
2016-05-18 00:57:55,324 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(199)) - Handle the remaining events in queue, queue size=0
2016-05-18 00:57:55,324 INFO recovery.RecoveryService (RecoveryService.java:run(173)) - EventQueue take interrupted. Returning
2016-05-18 00:57:55,324 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(221)) - Closing Summary Stream
2016-05-18 00:57:55,325 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-18 00:57:55,325 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-18 00:57:55,326 INFO rm.TaskSchedulerManager (TaskSchedulerManager.java:serviceStop(678)) - Shutting down AppCallbackExecutor
2016-05-18 00:57:55,329 INFO app.DAGAppMaster (DAGAppMaster.java:run(2084)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/tmp/.staging/E3CB2E9EB4474A08896DC26AA3629C50/A4F8FA78D9054C168AB5000C3B986FD6/.tez/application_1463533075047_0001
2016-05-18 00:57:55,329 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping all jobs
2016-05-18 00:57:55,329 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopping: (1/1) ...blyplatform/ALL_ARGS_SWAP
2016-05-18 00:57:55,330 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] stopped all jobs
2016-05-18 00:57:55,330 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output/hadoop2-tez/everypipeassemblyplatform/ALL_ARGS_SWAP/_temporary
2016-05-18 00:57:55,330 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [every] completed in: 00:00.301
2016-05-18 00:57:55,331 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.6.x-tez-0.8.x/build/test/output//hadoop2-tez/everypipeassemblyplatform//ALL_ARGS_SWAP